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Key findings 

This study examined academic outcomes for students enrolled in credit recovery programs 
offered by the North Carolina Virtual Public School (NCVPS) and other programs. 

NCVPS credit recovery students were less likely than other credit recovery students 
to be economically disadvantaged, and a greater proportion entered high school 
proficient in math and reading. 

There was little difference in short-term success rates (such as end-of-course exam 
scores) between NCVPS credit recovery students and other credit recovery students 
in the state. 

On measures of longer-term success, NCVPS credit recovery students were less 
likely to graduate than other credit recovery students, but those who did graduate 
were more likely to graduate on time (that is, within four years). 

Black NCVPS credit recovery students were less likely than students of other racial/ 
ethnic groups to reach proficiency in the recovered course (as measured by test 
scores) but were more likely to succeed in subsequent coursework in the same 
subject area after completion of the credit recovery course. 
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Summary 


Across the Regional Educational Laboratory Southeast Region there is growing interest in 
strengthening the presence of online learning in all public schools to help equalize edu- 
cation opportunities for all students and prepare students for a digital future. For instance, 
the North Carolina General Assembly has required that the state transition to digital 
learning tools by 2017, and work is under way to meet that goal. 

This study was designed to expand stakeholders’ understanding of one pre-existing aspect 
of digital learning that helped inspire the state’s transition — the extent to which online 
learning is already providing digitally enhanced options for students at risk of dropping out. 
Both virtual schools and state education agencies are interested in learning more about the 
reach of credit recovery programs (which allow students to retake required courses to make 
up graduation credits for courses they failed) and how outcome data differ across credit 
recovery options. These questions also are important to district-level personnel, especially 
in North Carolina. Though North Carolina dropout rates have decreased and graduation 
rates have risen in recent years, considerable public pressure remains for school districts to 
continue to improve these rates, and interest in using online credit recovery to address the 
issue is growing. 

This study examined the North Carolina Virtual Public School’s (NCVPS) credit recovery 
program (which was added to NC VPS’s extensive list of high school course offerings in 
2008) and other common credit recovery options available to students in the state (such as 
summer school and traditional school-year course repetition, as well as online credit recov- 
ery provided by third-party vendors). It also compared short- and longer-term academic 
outcome data across the credit recovery options. Finally, the study calculated correlations 
between the academic outcomes and characteristics of students enrolled in the various 
credit recovery options to lay the groundwork for future research on the efficacy of credit 
recovery programs. 

Key findings include: 

• NCVPS credit recovery students were less likely than other credit recovery stu- 
dents to be economically disadvantaged, and a greater proportion entered high 
school proficient in math and reading. 

• There was little difference in the short-term success rates (such as end-of-course 
exam scores) between NCVPS credit recovery students and other credit recovery 
students. 

• On measures of longer-term success (such as graduation rates), NCVPS credit recov- 
ery students were less likely than other credit recovery students to graduate, but those 
who did graduate were more likely to stay on track to graduate (by succeeding in 
subsequent related coursework) and to graduate on time (that is, within four years). 

• Black NCVPS credit recovery students were less likely than students of other 
racial/ethnic groups to reach proficiency in the recovered course (as measured by 
test scores) but were more likely to succeed in subsequent coursework in the same 
subject area after completion of the credit recovery course. 

Given the study’s design, the results of the analyses described in this report cannot be used 
to draw direct conclusions about the quality of online credit recovery courses or of their 
overall efficacy in improving student academic outcomes. 
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Why this study? 


This report complements earlier work conducted by Regional Educational Laboratory 
(REL) Southeast on Florida Virtual School credit recovery (Elughes, Zhou, & Petscher, 
2015) by extending that report’s discussion about the expanding role of online education 
in the REL Southeast Region to North Carolina. 

As is true across much of the REL Southeast Region, there is growing interest in North 
Carolina in strengthening the presence of online learning in all public schools. For 
example, in 2013 the North Carolina General Assembly directed the state to transition to 
digital learning tools by 2017 (North Carolina General Assembly, 2013), and the state is 
developing a longer-term comprehensive plan for incorporating digital learning into every 
school setting. This study expands understanding of two aspects of digital learning that 
helped inspire the state’s transition — the extent to which online learning already plays a 
role in providing digitally enhanced options for students at risk of dropping out, and the 
range and scope of un- or under-addressed needs of students in these online credit recovery 
programs. Previous research on this topic is limited (box 1). 

Two stakeholder audiences for this study are the North Carolina Virtual Public School 
(NCVPS), North Carolina’s provider of online courses, and the North Carolina Depart- 
ment of Public Instruction, the state’s education agency. To date, these organizations have 
analyzed only enrollment data related to student participation in NCVPS credit recovery; 
neither has conducted an in-depth analysis of student outcomes. This study was developed 
to respond to the need for such an analysis. In addition to expanding understanding of par- 
ticipation in and outcomes related to online credit recovery, this study also demonstrates 
the range of North Carolina online credit recovery data available for future analyses and 
how the data can be used. 

Viable credit recovery options are also of great interest to district-level personnel, who 
are directly responsible for deciding what academic options to make available to students. 
Though North Carolina dropout rates have declined and graduation rates have risen in 
recent years (see box 1), considerable public pressure remains for school districts to contin- 
ue to improve these rates. In addition to the state-provided online credit recovery option 
offered through NCVPS, school districts use traditional options such as school-year and 
summer school course repetition in a face-to-face classroom setting, as well as third-party 
online credit recovery programs, to improve graduation rates. This report addresses the 
need for better understanding of the similarities and differences across these programs 
(box 2). 

The analyses used in this study limit its ability to provide definitive conclusions about 
whether participation in online credit recovery leads directly to the short- and longer-term 
student outcomes examined — which include measures of student persistence in school and 
of student academic success. However, the results can focus future efforts on uncovering 
whether such causal relationships exist and, if so, how to maximize their potential. 


This study expands 
understanding of 
two aspects of 
digital learning in 
North Carolina- 
the extent to 
which online 
learning already 
plays a role in 
providing digitally 
enhanced options 
for students at risk 
of dropping out, 
and the range and 
scope of un- or 
under-addressed 
needs of students 
in these online 
credit recovery 
programs 
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Box 1. What other research has reported about student dropout and online credit 
recovery 

A decade after the No Child Left Behind law mandated efforts to reduce the achievement gap, 
nationwide about 87 percent of Asian and 84 percent of White students, but only 71 percent of 
Hispanic, 67 percent of Black, and 72 percent of economically disadvantaged students gradu- 
ate from high school (Stetser & Stillwell, 2014). Increasing graduation rates for all populations 
is important when considering the negative outcomes associated with the dropout population. 
The Center for Labor Market Studies (2009) estimates that each adult with a high school 
diploma has the potential to contribute at least $250,000 more than high school dropouts as 
a result of combined lifetime fiscal benefits (via taxes), and to lower the social costs associat- 
ed with incarceration, welfare, and healthcare. 

Despite sharply increased graduation rates in recent years, North Carolina’s graduation 
rate has only recently climbed to the national average: four-year cohort graduation rates rose 
from 68.3 percent in 2006 to 83.9 percent in 2014 (North Carolina Department of Public 
Instruction, 2015a), surpassing the national average of 82 percent that year. And despite 
2014’s highest graduation rate on record for the state, the total number of high school drop- 
outs (10,404) also remained high (North Carolina Department of Public Instruction, 2015b). 
This means that student dropout and its associated costs remain an important issue in North 
Carolina. 

There is growing consensus among researchers that dropping out of school is not a static 
or isolated event but is instead the final event in a long-term process of disengagement from 
school that often involves multiple factors, many of which may have originated as early as 
elementary school. This theorized cumulative process of disengagement from school suggests 
that if the appropriate resources are implemented at the right time, students can be re-en- 
gaged and get back on track to graduate (Alexander, Entwisle, & Horsey, 1997; Ensminger 
& Slusarcick, 1992; Fine, 1991; Finn, 1993; McNeal, 1997; Rumberger, 2001; U.S. General 
Accounting Office, 2002; Vitaro, Brendgen, & Tremblay, 1999). 

Online credit recovery programs are one way in which states have attempted to help stu- 
dents stay on track. Benefits of virtual courses are well documented for general and honors 
students in addressing issues such as access, student achievement, and twenty-first-century 
learning skills (Cavanaugh, Gillan, Kromrey, Hess, & Blomeyer, 2004; Means, Toyama, Murphy, 
Bakia, & Jones, 2009); however, there is little rigorous research about the experience and 
performance of low-achieving students in online learning environments (Cavanaugh, Barbour, & 
Clark, 2009). There is growing qualitative evidence that credit recovery programs (face to face 
as well as virtual) may improve graduation rates (Christian, 2003; Watson & Gemin, 2008; 
Dessoff, 2009; Menzer & Hampel, 2009), but few empirical studies support these findings 
(Zehr, 2010). Of the few quantitative studies that have examined credit recovery programs and 
short-term outcomes (such as pass rates and course grades), results are mixed, with some 
studies (such as Hughes et al., 2015) suggesting that online credit recovery courses may help 
students earn better grades than those earned by students taking the same recovery courses 
face to face. Others (such as Heppen et al., 2013) offer evidence that face-to-face credit 
recovery courses may benefit students more than online courses in terms of higher grades and 
pass rates. 
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Box 2. Credit recovery options in North Carolina 


Credit recovery. Retaking a previously failed course required for graduation. For example, a 
student who needs English I to graduate, but fails the course, may retake English I at a later 
date to “recover” that credit. 

North Carolina Virtual Public School credit recovery program. The state-provided online credit 
recovery option for students who fail high school courses required for graduation. 

Third-party credit recovery programs. Another online credit recovery option in North Carolina. 
Several school districts enroll students in online credit recovery programs offered by third-par- 
ty vendors. Third-party credit recovery program vendors operating in North Carolina for the 
years covered in this study were Ed Options, NovaNET, Novel Stars, Odyssey Ware, Plato, SAS 
Curriculum Pathways, and Study Island. 

Traditional credit recovery options. Courses retaken in a traditional, face-to-face classroom 
setting. This study examined student enrollment in two types of traditional credit recovery 
options: 

• School-year course repetition refers to courses retaken in a traditional classroom setting 
during the fall or spring semester. 

• Summer school course repetition refers to courses retaken in a traditional classroom 
setting during the summer. 


What the study examined 


This study addressed three primary research questions. 

1. What are the features of and enrollment in NCVPS credit recovery courses and the 
other credit recovery programs available to students in North Carolina, and what 
are the characteristics of students who enroll in NCVPS and other credit recovery 
options? 

2. How do short-term success rates (as measured by scores on end-of-course exam retests) 
for students who complete NCVPS credit recovery courses compare with success rates 
for students who repeat a course through other credit recovery options? 

3. After completing an NCVPS credit recovery course, how do mid- to longer-term 
success rates (such as staying on track and graduating) compare with success rates for 
students who repeat a course through any other credit recovery option? 

As an extension to research questions 2 and 3, the study also analyzed outcomes for student 

subgroups within the NCVPS credit recovery sample: 

4. To what extent, if any, do success rates in subsequent exams, courses, dropout rates, 
and graduation rates for NCVPS credit recovery students vary by student subgroup? 

For this extension the study team examined whether any correlations exist between 

the student outcomes highlighted in research questions 2 and 3 and three observable 
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characteristics of NCVPS credit recovery students: whether a student is economically dis- 
advantaged (as proxied by eligibility for the federal school lunch program), a student’s race/ 
ethnicity, and the number of end-of-course exams a student failed in the academic year of 
her or his initial course failure. Box 3 summarizes the data and analyses used; appendix A 
provides more details. 

The report includes a descriptive inventory of the credit recovery options available to 
high school students from fall 2008 through summer 2012 in North Carolina. 1 In addition 
to NCVPS credit recovery, other credit recovery options detailed in this report include 
courses retaken in a traditional, face-to-face classroom setting (either during the school 
year or in summer school) and online programs provided by third-party credit recovery 
program vendors. The report summarizes characteristics of students enrolled in NCVPS 
credit recovery and the two main traditional credit recovery options for which reliable 


Box 3. Data, methods, and interpretation 

Data. This study used data from North Carolina Virtual Public School (NCVPS), the North Car- 
olina Department of Public Instruction, and third-party credit recovery programs. NCVPS pro- 
vided credit recovery program and student enrollment data, the North Carolina Department of 
Public Instruction provided student records that included individual demographic and academic 
variables, and the seven third-party credit recovery program vendors in operation in the state 
provided descriptions of their programs. 

The study included students who took credit recovery courses offered by NCVPS between 
2008/09 — the first full academic year of NCVPS’s program — and 2011/12, as well as all 
identifiable students whose credit recovery involved re-enrolling in a course during the regular 
school year (traditional face-to-face credit recovery), during summer school, or as part of some 
other credit recovery program (see appendix A for details about the data and methods used, 
and box A1 for details about efforts to identify participants in other credit recovery options). 

Methods Each section of this report leads with simple descriptive statistics (unadjusted sum- 
maries of outcomes for different student subgroups). The study also used regression analy- 
ses, which partially factor out the combined influence of multiple variables on an outcome of 
interest to provide a clearer sense of the relationship between the outcome and each variable 
independent of all other variables. The study relies on two types of regression analyses: analy- 
ses of outcomes with many potential values and analyses of outcomes with only two potential 
values. 

Interpretations of results. 

• Interpreting effect sizes. When the results of a regression analysis with many potential 
outcomes are statistically significant (that is, not likely to have happened by chance), there 
is a temptation to assume that the results are meaningful. What is often more important 
is the size of the predicted result. The effect size helps put into context how meaningful a 
statistically significant result is. This study considers effect sizes between 0.2 and 0.5 to 
be small, effect sizes from 0.5 to 0.8 to be medium, and effect sizes of 0.8 and above to 
be large (Cohen, 1988). 

• Interpreting likelihood. When there are only two possible outcomes of a regression analysis 
(0 or 1, yes or no), likelihood is used instead of effect sizes to estimate the feasibility of 
an outcome for student subgroups. In this study the odds of an event happening for one 
group is calculated relative to the odds of the event happening for a comparison group. 
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student demographic data were available (schoohyear and summer school course repeti- 
tion 2 ), including grade level, race, gender, economic disadvantage status, special education 
status, grade 8 end-of-grade exam scores, and other measures. 

The report compares short-term education outcomes for NCVPS credit recovery students 
with outcomes for students who participated in credit recovery through providers other 
than NCVPS. These short-term outcomes included students’ standardized state end-of- 
course exam scores and proficiency levels following credit recovery. The report examines 
four longer-term education outcomes: whether a student re-enrolled in school the follow- 
ing year, reached proficiency in the next course in the course series, graduated from high 
school, and graduated from high school on time (that is, within four years). 


What the study found 


Findings are reported for four broad categories of outcomes: information about online and 
face-to-face credit recovery options in North Carolina, comparisons of student characteris- 
tics by credit recovery option, measures of short- and longer-term outcomes for students by 
credit recovery option, and differences in outcomes for subgroups of NCVPS credit recov- 
ery students. 

Traditional, school-year course repetition remains the most common credit recovery option in North 
Carolina, but online and summer school credit recovery enrollments are growing fast 

This section provides detailed information about each of the credit recovery options avail- 
able in North Carolina, as well as historical information about enrollment in each. 


In the 2011/12 
school year (the 
last year in the 
current study) the 
North Carolina 
Virtual Public 
School program 
served students in 
101 (88 percent) 
of the state’s 115 
school districts, as 
well as students 
in charter schools 
and federal schools 


NCVPS credit recovery. NCVPS was established by the North Carolina E-Learning 
Commission in 2005 and began operations in 2007. NCVPS grew quickly; by 2015 it 
offered more than 150 courses and enrolled more than 52,000 students in courses ranging 
from Advanced Placement and other college credit courses to honors and general courses 
in English, math, science, social studies, world languages, arts, career and technical edu- 
cation, and healthful living. NCVPS also offers test preparation and career planning serv- 
ices. Course offerings are available to middle and high school students. 

In January 2008 NCVPS introduced online credit recovery courses for students who failed 
high school courses required for graduation. Since its inception, NCVPS credit recovery 
has grown into a fully developed program. In the 2011/12 school year (the last year in the 
current study) the program served students in 101 (88 percent) of the state’s 115 school 
districts, as well as students in charter schools and federal schools. 

• Courses offered. The credit recovery courses in which students enrolled between 
2008/09 and 2011/12 included English I, II, III, and IV; Algebra I, Algebra II, and 
Geometry; Physical Science and Biology; and Civics and Economics, U.S. Elistory, 
and World History. 

• Coverage. All NCVPS credit recovery courses align to state curriculum standards; 
use a self-paced, mastery-learning model; are written and taught by highly qual- 
ified North Carolina teachers trained to teach online; contain all instructional 
materials needed for the student and require no modifications by local school 
districts; and are available to all high school students enrolled in participating 
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North Carolina public schools, charter schools, Department of Defense schools, 
and schools operated by the Bureau of Indian Affairs. 

• Length of student participation. Most students completed the courses during one 
semester of the school year or during an eight-week summer session. 

• Student autonomy. NCVPS credit recovery courses are designed to be student cen- 
tered. Once a student masters an assignment, the student can move on to the next 
assignment. Once all course units are mastered, NCVPS notifies the student’s 
school of his or her completion of the course. 

• Format. NCVPS courses use an online learning management system that relies on 
interactive whiteboards, wikis, virtual environments, and online discussion tools. 
While the exact setting can vary by school district, a typical student who enrolled 
in a NCVPS credit recovery course during the study period attended a physical 
class at his or her school in a computer lab with other students who were taking 
other NCVPS courses — not all of which were credit recovery courses. In addition 
to the online course instructor provided by NCVPS, a teacher or teacher assis- 
tant typically was present to supervise the lab. The course format supported syn- 
chronous and asynchronous communication. Students had weekly contact with 
their teacher in a synchronous environment, as well as daily asynchronous contact 
through the messaging system. 

Traditional credit recovery options: school-year and summer school course repetition. 
Courses retaken in a traditional, face-to-face classroom setting (either during the school 
year or in summer school) are the most common credit recovery options available to North 
Carolina public school students. Implementation of these options (such as which courses 
are offered, how long students must participate to earn credit, cost per student, and so on) 
varies across school districts and across school years, but in almost all cases the courses are 
not mastery based, and completion is based at least in part on seat time. Overall, enroll- 
ment in traditional, school-year credit recovery courses declined slightly over the four 
years of the study, with grade 9 enrollment showing the largest drop from nearly 15,500 in 
2008/09 to just over 11,500 in 2011/12. In contrast, summer school enrollment in nearly all 
courses increased slightly over the four years covered in the study. 

While traditional, school-year course repetition continues to be the most frequently used 
credit recovery option in North Carolina, both traditional summer school and NCVPS 
credit recovery options have made enrollment gains in recent years (tables 1-3). 

Other online credit recovery options in North Carolina. As noted above, NCVPS credit 
recovery is not the only online credit recovery option in North Carolina. Several school 
districts also enroll students in online credit recovery programs offered by third-par- 
ty vendors. Because the choice to use third-party programs predated the availability of 
NCVPS credit recovery courses, some school districts retain relationships with those 
programs. Also, a change in the NCVPS funding formula that shifted more costs to the 
district starting in the 2011/12 school year has reduced the likelihood that NCVPS enroll- 
ment will overtake all other online options. 

• Courses offered. The third-party vendors offered a variety of online content that 
ranged from supplemental resources (such as web-based lessons and interactive 
tools) to full courses. Use of most third-party products was not restricted to credit 
recovery, and many third-party vendors indicated that each school district estab- 
lished guidelines for how a vendor’s products could be used. 3 As a result of this 


Overall, enrollment 
in traditional, 
school-year credit 
recovery courses 
declined slightly 
over the four years 
of the study, with 
grade 9 enrollment 
showing the largest 
drop from nearly 
15,500 in 2008/09 
to just over 11,500 
in 2011/12. in 
contrast, summer 
school enrollment 
in nearly all 
courses increased 
slightly over the 
four years covered 
in the study 
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Table 1. Student enrollment in North Carolina credit recovery courses, by grade 
level and credit recovery option, 2008/09-2011/12 

Grade level and credit recovery option 

2008/09 

2009/10 

2010/11 

2011/12 

Grade 9 

NCVPS credit recovery 

420 

787 

904 

654 

Traditional, school-year course repetition 

15,457 

13,216 

11,879 

11,543 

Traditional, summer school course repetition 

1,728 

1,949 

2,171 

2,621 

Grade 10 

NCVPS credit recovery 

653 

1071 

1172 

985 

Traditional, school-year course repetition 

18,065 

17,529 

15,031 

15,606 

Traditional, summer school course repetition 

1,837 

2,283 

2,659 

3,138 

Grade 11 

NCVPS credit recovery 

663 

1056 

1104 

1077 

Traditional, school-year course repetition 

15,794 

16,132 

13,391 

14,041 

Traditional, summer school course repetition 

1,950 

2,214 

2,272 

2,928 

Grade 12 

NCVPS credit recovery 

187 

505 

509 

491 

Traditional, school-year course repetition 

11,349 

13,421 

10,531 

10,542 

Traditional, summer school course repetition 

495 

531 

518 

460 


NCVPS is North Carolina Virtual Public School. 


Note: Because of missing data or errors in the administrative data related to students' grade level, enrollment 
totals in tables 1, 2, and 3 do not match. Traditional school year and summer school credit recovery courses 
refer to face-to-face courses. 

Source: Authors' analysis of transcript data provided by the North Carolina Virtual Public School and the Edu- 
cation Research Data Center. 


variability in how districts used their products, most vendors had limited ability 
to track or differentiate exactly how their content was being used by a teacher or 
student (such as whether it was used for credit recovery or for some other purpose). 
Though unable to count credit recovery students precisely, vendors indicated that 
many schools or districts that contracted with them did so specifically for the 
purpose of online credit recovery. Also, some vendors offered specially designed 
versions of their courses for credit recovery that included pretest assessments to 
customize lessons for individual students. All vendors noted that the most com- 
monly subscribed courses or materials were for English and math. See box A1 in 
appendix A for a detailed explanation of how third-party credit recovery courses 
were included in this study’s analyses. 

• Length of student participation. Across all third-party vendors, online courses were 
typically designed to be completed within a summer, semester, or yearlong time- 
frame. Due to the nature of many of these courses (self-paced, asynchronous, and 
the like) students often had flexibility to complete a given course in a shorter time- 
frame if they desired. Course durations could also be extended beyond the stan- 
dard timeframe for a student, pending district approval. Most third-party vendors 
noted that each district set its own rules for credit recovery course completion 
requirements. 

• Student autonomy. All available courses across vendors were designed to be student 
driven. While students could advance at their own pace, it was typical for a face- 
to-face teacher to assist students when necessary. No third-party vendor provided 
online teachers. All vendor representatives emphasized that districts decide how 


Across all third- 
party vendors, 
online courses 
were typically 
designed to be 
completed within a 
summer, semester, 
or yearlong 
timeframe, and 
students often 
had flexibility 
to complete a 
given course in a 
shorter timeframe 
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Table 2. Student enrollment in North Carolina credit recovery courses with an end- 
of-course exam, by course and credit recovery option, 2008/09-2011/12 

Course and credit recovery option 

2008/09 

2009/10 

2010/11 

2011/12 

English 1 

NCVPS credit recovery 

272 

450 

409 

278 

Traditional, school-year course repetition 

8,912 

7,506 

6,929 

7,116 

Traditional, summer school course repetition 

705 

744 

708 

888 

Algebra 1 

NCVPS credit recovery 

361 

484 

435 

435 

Traditional, school-year course repetition 

18,518 

15,853 

14,931 

13,306 

Traditional, summer school course repetition 

990 

976 

1,123 

1,284 

Algebra II 

NCVPS credit recovery 

212 

373 

374 

483 

Traditional, school-year course repetition 

5,101 

3,601 

3,458 

5,879 

Traditional, summer school course repetition 

436 

419 

662 

1,194 

Geometry 

NCVPS credit recovery 

213 

320 

482 

556 

Traditional, school-year course repetition 

4,372 

3,890 

4,475 

7,994 

Traditional, summer school course repetition 

480 

611 

1,012 

1,490 

Physical Science 

NCVPS credit recovery 

127 

212 

167 

132 

Traditional, school-year course repetition 

3,330 

3,255 

3,004 

3,099 

Traditional, summer school course repetition 

258 

323 

284 

366 

Biology 

NCVPS credit recovery 

0 

138 

333 

217 

Traditional, school-year course repetition 

6,875 

6,941 

6,332 

6,004 

Traditional, summer school course repetition 

501 

711 

629 

554 

Civics and Economics 

NCVPS credit recovery 

241 

442 

363 

324 

Traditional, school-year course repetition 

6,680 

6,764 

6,612 

6,171 

Traditional, summer school course repetition 

479 

605 

477 

731 

U.S. History 

NCVPS credit recovery 

187 

273 

304 

279 

Traditional, school-year course repetition 

4,776 

4,152 

4,190 

3,825 

Traditional, summer school course repetition 

508 

471 

484 

719 


NCVPS is North Carolina Virtual Public School. 

Note: Bolding indicates courses and years in which North Carolina end-of-course exams were administered; 
end-of-course exams were discontinued for some courses during the study period. Because of missing data or 
errors in the administrative data related to students’ grade level, enrollment totals in tables 1, 2, and 3 do not 
match. Traditional school year and summer school credit recovery courses refer to face-to-face courses. 

Source: Authors’ analysis of transcript data provided by the North Carolina Virtual Public School and the Edu- 
cation Research Data Center. 
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Table 3. Student enrollment in North Carolina credit recovery courses without an 
end-of-course exam, by course and credit recovery option, 2008/09-2011/12 

Course and credit recovery option 

2008/09 

2009/10 

2010/11 

2011/12 

English II 

NCVPS credit recovery 

282 

547 

556 

433 

Traditional, school-year course repetition 

9,243 

8,881 

6,432 

6,691 

Traditional, summer school course repetition 

788 

1,015 

1,137 

1,273 

English III 

NCVPS credit recovery 

288 

524 

543 

397 

Traditional, school-year course repetition 

7,617 

7,351 

5,260 

5,402 

Traditional, summer school course repetition 

926 

1,014 

948 

894 

English IV 

NCVPS credit recovery 

85 

131 

160 

111 

Traditional, school-year course repetition 

4,747 

4,207 

1,987 

2,064 

Traditional, summer school course repetition 

270 

297 

329 

277 

World History 

NCVPS credit recovery 

0 

230 

513 

230 

Traditional, school-year course repetition 

13,027 

12,845 

10,054 

9,198 

Traditional, summer school course repetition 

615 

716 

787 

841 


NCVPS is North Carolina Virtual Public School. 


Note: Because of missing data or errors in the administrative data related to students' grade level, enrollment 
totals in tables 1, 2, and 3 do not match. Traditional school year and summer school credit recovery courses 
refer to face-to-face courses. 

Source: Authors' analysis of transcript data provided by the North Carolina Virtual Public School and the Edu- 
cation Research Data Center. 


the content was delivered to students; thus, it was possible for courses to be entire- 
ly student-driven and completed online or to be delivered or administered by a 
face-to-face teacher. 

• Format. Across vendors, course content or supplemental resources were designed 
primarily to be completed asynchronously; however, many courses included ele- 
ments that supported synchronous communication or activities (that is, message 
boards, group activities, and the like). 

• Coverage. Of the seven third-party vendors contacted, only one was willing and 
able to share the names of the districts with which it contracted and the number 
of students in each district who used the company’s product during the time period 
of this study. The other vendors cited individual district contracts that prohibited 
them from sharing this type of information, or an inability to share detailed cov- 
erage data due to limitations in data-tracking and management processes. Based 
on the data available at the time, third-party credit recovery program vendors 
appeared to have operated in at least 100 of North Carolina’s 115 districts between 
2008/09 and 2011/12. 


Across vendors, 
course content 
or supplemental 
resources were 
designed primarily 
to be completed 
asynchronously; 
however, many 
courses included 
elements that 
supported 
synchronous 
communication 
or activities 


Some districts periodically have offered locally developed programs, but these are not 
clearly identified and are not considered separately in this study. 
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North Carolina Virtual Public School students were less likely than their peers in traditional credit 
recovery programs to be economically disadvantaged, and a greater proportion entered high school 
proficient in math and reading 


Another important consideration when comparing credit recovery options is whether 
student populations are similar. While NCVPS credit recovery students were similar to 
other credit recovery students in many ways, there were some notable differences across 
student groups. 


Qender. Across all four years of the study and across credit recovery options, enrollment 
by gender for each credit recovery option remained fairly constant, with a higher propor- 
tion of male students (56-64 percent) than female students (37-44 percent) enrolled in 
courses. Overall, female student enrollment across credit recovery options declined slight- 
ly, while male student enrollment increased slightly. Differences in enrollment by gender 
across credit recovery options were negligible, but female student enrollment was typically 
highest for NCVPS credit recovery and typically lowest for summer school. 

Race/ethnicity. Across credit recovery options and across years, Black students account- 
ed for a disproportionately large proportion of enrollment (41-50 percent; table 4). The 
proportion of Black students across the state between 2008/09 and 2011/12 was about 
28 percent (North Carolina Department of Public Instruction, n.d. b.). However, it is 
outside the scope of the study to speculate about reasons for this discrepancy. Though 
most differences in enrollment across credit recovery options were not large within racial/ 


Table 4. Student race/ethnicity in North Carolina, by credit recovery option, 
2008/09-2011/12 (percent) 

Race/ethnicity and credit recovery option 

2008/09 

2009/10 

2010/11 

2011/12 

Black 

NCVPS credit recovery 

45.3 

44.1 

42.3 

41.6 

Traditional, school-year course repetition 

40.6 

43.0 

43.6 

42.0 

Traditional, summer school course repetition 

48.3 

49.7 

50.1 

48.3 

Hispanic 

NCVPS credit recovery 

6.0 

8.6 

11.2 

13.7 

Traditional, school-year course repetition 

9.8 

10.9 

12.7 

13.9 

Traditional, summer school course repetition 

7.5 

7.3 

10.3 

12.3 

Other 

NCVPS credit recovery 

4.2 

4.3 

4.5 

6.1 

Traditional, school-year course repetition 

5.5 

5.4 

6.2 

6.4 

Traditional, summer school course repetition 

4.2 

5.1 

5.4 

5.5 

White 

NCVPS credit recovery 

44.5 

43.1 

40.1 

38.1 

Traditional, school-year course repetition 

44.0 

40.7 

37.6 

37.7 

Traditional, summer school course repetition 

39.8 

38.0 

34.2 

33.9 


NCVPS is North Carolina Virtual Public School. 

Note: Black includes African American. Hispanic includes Latino. Other includes Asian, American Indian/ 
Alaskan Native, Hawaiian Native/Pacific Islander, and multiracial. Traditional school year and summer school 
credit recovery courses refer to face-to-face courses. 

Source: Authors’ analysis of administrative data provided by the Education Research Data Center. 


Across all four 
years of the study 
and across credit 
recovery options, 
enrollment by 
gender for each 
credit recovery 
option remained 
fairly constant, 
with a higher 
proportion of male 
students (56- 
64 percent) than 
female students 
(37-44 percent) 
enrolled 
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ethnic groups, among White students summer school typically had the lowest enrollment 
proportion and NCVPS credit recovery the highest enrollment proportion. 

Economic disadvantage status. Matching a statewide trend, the proportion of credit 
recovery students who were economically disadvantaged increased steadily over the four 
years of the study; however, among credit recovery options the proportion was lower 
for NCVPS than for traditional schools and summer school in every year after 2008/09 
(table 5). 


Students receiving special education services. The proportion of students receiving 
special education services was similar across credit recovery options. Traditional school- 
year and NCVPS enrollment of students receiving these services ranged from 14 percent 
to 18 percent across years, and summer school enrollment ranged from 16 percent to 
17 percent. 

Qrade 8 end-of-grade exam proficiency. Changes in performance on state-administered 
end-of-grade exams across years are not surprising in North Carolina, as end-of-grade scores 
typically track upward until the exam form is changed. More important are differences 
across credit recovery options in a given year, and for the three most recent years in math 
and the two most recent years in reading, NCVPS credit recovery students tended to earn 
higher average proficiency ratings than did traditional credit recovery students (table 6). 


The proportion of 
credit recovery 
students who 
were economically 
disadvantaged 
increased steadily 
over the four years 
of the study 


Table 5. Economically disadvantaged students in North Carolina, by credit recovery 
option, 2008/09-2011/12 (percent) 


1 Credit recovery option 

2008/09 

2009/10 

2010/11 

2011/12 1 

NCVPS credit recovery 

47.6 

51.9 

52.1 

55.9 

Traditional, school year course repetition 

47.3 

56.7 

64.0 

67.5 

Traditional, summer school course repetition 

50.0 

57.0 

60.3 

63.9 


NCVPS is North Carolina Virtual Public School. 

Note: Traditional school year and summer school credit recovery courses refer to face-to-face courses. 
Source: Authors' analysis of administrative data provided by the Education Research Data Center. 


Table 6. Grade 8 North Carolina end-of-grade exam proficiency of students, by 
content area and credit recovery option, 2008/09-2011/12 (percent) 



NCVPS credit recovery 48.0 53.3 58.0 64.1 

Traditional, school-year course repetition 53.2 47.8 49.0 57.7 

Traditional, summer school course repetition 45.2 48.5 57.9 63.7 

Reading 

NCVPS credit recovery 77.6 66.6 52.7 49.2 

Traditional, school-year course repetition 80.2 70.5 47.6 41.0 

Traditional, summer school course repetition 69.9 55.5 43.3 45.1 

NCVPS is North Carolina Virtual Public School. 

Note: Traditional school year and summer school credit recovery courses refer to face-to-face courses. 
Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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North Carolina Virtual Public School students scored lower on state end-of-course exam retests after 
credit recovery than did traditional credit recovery students, but the differences were small once 
differences between student subgroups were taken into account 


This part of the study compared two short-term academic outcomes for NCVPS credit 
recovery students — scaled scores and achievement levels on standardized North Carolina 
end-of-course exams retaken after completion of credit recovery — to outcomes for all other 
credit recovery students (traditional face-to-face students, summer school students, and 
other identifiable credit recovery students, such as those who participated in third-party 
credit recovery options; see appendix A). 


The study team examined end-of-course exam retests because students took them after 
completing their credit recovery courses. At the individual course level and across all 
courses, the NCVPS credit recovery students in the sample years reached proficiency on 
their end-of-course exam retests at lower rates (0.3-19.6 percentage points) and scored 
lower on their end-of-course exam retests after credit recovery (0.1-2. 6 points) than did 
other credit recovery students (table 7). 

Even after the analysis adjusted for other possible influences that might be related to these 
differences in retest scores (such as credit recovery option, student demographics, and prior 
scores on the same exam; see appendix A for full regression equations), NCVPS credit 
recovery students still appeared to be less likely to score as high as or to reach proficiency 


Table 7. Performance on North Carolina end-of-course exam retest after completing 
credit recovery, 2008/09-2011/12 

Outcome and course 

NCVPS credit 
recovery students 

All other credit 
recovery students 

Difference in 
percent proficient 

Students demonstrating proficiency on end-of-course exam retest (percent) 

English 1 

34.6 

34.9 

-0.3 

Algebra 1 

29.5 

36.1 

-5.6 

Algebra II 

45.5 

55.6 

-10.1 

Geometry 

29.9 

49.5 

-19.6 

Physical Science 

30.8 

45.0 

-14.2 

Biology 

26.9 

41.2 

-14.3 

Civics and Economics 

27.1 

36.0 

-8.9 

U.S. History 

31.7 

43.1 

-11.4 

Average scaled score on end-of-course exam retest 

English 1 

142.8 

142.9 

-0.1 

Algebra 1 

143.3 

144.7 

-1.4 

Algebra II 

145.1 

146.9 

-1.8 

Geometry 

144.2 

146.8 

-2.6 

Physical Science 

144.3 

146.8 

-2.5 

Biology 

142.3 

144.6 

-2.3 

Civics and Economics 

144.0 

145.3 

-1.3 

U.S. History 

145.7 

147.0 

-1.3 


At the individual 
course level and 
across all courses, 
the NCVPS credit 
recovery students 
in the sample 
years reached 
proficiency on 
their end-of-course 
exam retests 
at lower rates (0.3- 
19.6 percentage 
points) and scored 
lower on their end- 
of-course exam 
retests after credit 
recovery (0. 1-2.6 
points) than 
did other credit 
recovery students 


NCVPS is North Carolina Virtual Public School. 

Note: Traditional school year and summer school credit recovery courses refer to face-to-face courses. 
Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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as often as other credit recovery students after completing their credit recovery course (see 
table B1 in appendix B and appendix C for more detailed results). 


The differences revealed by these more complex, exploratory analyses were statistically 
significant, but because the effect sizes were very small (all well below 0.2), the differences 
were not particularly meaningful (see table B1 in appendix B and box 3 for an explanation 
of effect size). 4 In other words, while there was no indication that participation in NCVPS 
credit recovery might have been more beneficial to credit recovery students in terms of 
shorter-term outcomes than participation in other credit recovery options, there also was 
no definitive indication, based on the differences in the statistically adjusted changes in 
scaled scores before and after credit recovery, that participation might have been notably 
detrimental. As noted elsewhere in this report, it is not possible to make definitive infer- 
ences of causality based on these analyses, because the differences in outcomes may reflect 
differences between the pre-existing characteristics of participants in NCVPS and other 
credit recovery programs rather than the effects of the credit recovery programs themselves. 

Longer-term outcomes for North Carolina Virtual Public School credit recovery students were mixed 
relative to outcomes for other credit recovery students 

Next, the study compared four longer-term binary (yes/no) education outcomes for NCVPS 
credit recovery students and for other credit recovery students: 

• Student re-enrolled in school the following year. 

• Student reached proficiency in the subsequent course in the course series — an 
indicator derived by stringing together the sequence of students’ courses in their 
longitudinal records. 

• Student graduated from high school. 

• Student graduated from high school on time (that is, within four years). 


Between 2008/09 
and 2011/12, 
NCVPS credit 
recovery students 
re-enrolled 
at a rate of 

91.7 percent, 

2.7 percentage 
points higher 
than other credit 
recovery students 


Re-enrollment and subsequent-course proficiency. A critical first step in the graduation 
process for students who fail a class is successfully getting back on track for graduation, 
first through re-enrollment in school and then through success in courses immediately 
after re-enrollment. Between 2008/09 and 2011/12, NCVPS credit recovery students re- 
enrolled at a rate of 91.7 percent, 2.7 percentage points higher than other credit recovery 
students. NCVPS credit recovery students’ academic success rates in the next course in 
each sequence (for example, passing English II after taking English I credit recovery), rela- 
tive to students who completed other credit recovery options, varied between 3.9 percent- 
age points lower and 2.4 percentage points higher (table 8). 


Regression analysis was then applied to determine whether these results would hold up 
after adjusting for other possible explanatory factors (such as differences in special educa- 
tion status and school attendance rates; see box 3). NCVPS credit recovery students still 
appeared to be more likely than their peers to re-enroll in school the year following their 
credit recovery coursework (see table B2 in appendix B and table C26 in appendix C), but 
the adjustments did eliminate the two positive results for English course sequences report- 
ed in table 8 (see table B2 in appendix B and tables C21-C25 in appendix C for specific 
results). 


13 


Table 8. Longer-term in-school outcomes in North Carolina after completing credit 


recovery, 2008/09-2011/12 (percent) 

Outcome 

NCVPS credit 
recovery 
students 

All other 
credit recovery 
students 

Difference 
in percent 
achieving 

Student re-enrolls following year 

91.7 

89.0 

+2.7 

Student passes next course in sequence 

English II after English 1 credit recovery 

67.1 

68.0 

-0.9 

English III after English II credit recovery 

73.6 

71.2 

+2.4 

English IV after English III credit recovery 

88.4 

86.9 

+1.5 

Algebra II or Geometry, after Algebra 1 credit recovery 

60.8 

64.7 

-3.9 

NCVPS is North Carolina Virtual Public School. 


Source: Authors' analysis of administrative data provided by the Education Research Data Center. 


Qraduation. On the ultimate measure of secondary schooHevel longer-term outcomes — 
graduation — NCVPS credit recovery students in the sample years graduated at a lower rate 
(76.0 percent) than did other credit recovery students (77.7 percent; table 9). 

Even after the analysis controlled for several variables through regression analyses (see box 
3), NCVPS credit recovery students still appeared to be less likely than their credit recovery 
peers to graduate, a difference that was statistically significant. However, among those who did 
graduate, they were more likely than their peers to graduate on time (that is, within four years 
of entering high school; see table B3 in appendix B and tables C27 and C28 in appendix C). 

Taken together, these findings suggest potentially useful areas for further research. In 
particular, there appears to be value in exploring further not only whether the flexibility 
of the state’s online credit recovery option — which may be easier to schedule not only 
during the regular school day but also potentially outside of it, and which can more readily 
allow for rapid make-up of coursework — may be an effective tool for helping some students 
stay enrolled and on track for on-time graduation, but also whether those possible gains 
are offset by other apparent losses in academic performance and completion. However, 
because of the nature of the current study and because the comparison group includes a 
small number of third-party online credit recovery students, these analyses only allowed 
the study team to speculate, not to confirm, without the benefit of a more rigorous and 
controlled study. It may be just as likely that there are other factors that are systematically 
similar about NCVPS credit recovery students that increases their likelihood to exhibit 
some characteristics of academic persistence but that the demographics data used in these 
analyses cannot uncover. 


NCVPS credit 
recovery students 
in the sample 
years graduated 
at a lower rate 
(76.0 percent) than 
did other credit 
recovery students 
(77.7 percent); 
however, among 
those who did 
graduate, they 
were more likely 
than their peers to 
graduate on time 


Table 9. Longer-term graduation outcomes in North Carolina after completing credit 
recovery, 2008/09-2011/12 (percent) 


Outcome 

NCVPS credit 
recovery 
students 

All other 
credit recovery 
students 

Difference 
in percent 
achieving 

Student graduates 

76.0 

77.7 

-1.7 

If student graduates, graduates on time 

(within four years) 

31.5 

24.3 

+7.2 


NCVPS is North Carolina Virtual Public School. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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Among North Carolina Virtual Public School credit recovery students, Black students were less likely 
to reach proficiency but more likely to succeed in later coursework 


Finally, this study examined the outcomes for research questions 2 and 3 within the pop- 
ulation of NCVPS credit recovery students only, by considering both raw outcomes and 
outcomes analyzed through regression (see box 3) to estimate whether the outcomes differ 
after controlling for other possible explanatory variables. The goals were first to note dif- 
ferences between outcomes for different student subgroups and then to identify possible 
correlations between these outcomes and three key student characteristics: 

• A student’s race/ethnicity. 

• Whether a student was economically disadvantaged. 

• The number of end-of-course exams a student failed in the academic year in which 
she or he first failed a course (a more nuanced indicator than the raw number of 
courses failed of differences in the level of academic challenge that a given student 
faces after failing for the first time). 

Race/ethnicity. Racial/ethnic minority NCVPS credit recovery students frequently had 
higher success rates than White students, not only in terms of re'enrolling the year follow- 
ing credit recovery, but also in terms of graduating and graduating on time. For example, 
White NCVPS credit recovery students re-enrolled 85.1 percent of the time, compared with 
89.4 percent of the time for Black NCVPS credit recovery students and 89.5 percent of the 
time for Hispanic NCVPS credit recovery students. White NCVPS credit recovery stu- 
dents graduated 73.5 percent of the time, compared with 78.8 percent of the time for Black 
NCVPS credit recovery students and 83.7 percent for Asian NCVPS students (table 10). 

After analyses controlled for the influence of other explanatory variables (such as differ- 
ences in special education status and school attendance rates) through regression, racial/ 
ethnic minority NCVPS credit recovery students still appear to have been more likely 
than White NCVPS credit recovery students to succeed on all three longer-term outcome 
measures (see tables C29-C31 in appendix C for detailed results). However, there were few 
statistically significant and meaningful patterns across races/ethnicities for improvements 
in exam scores after credit recovery. 


White NCVPS 
credit recovery 
students 
re-enrolled 
85.1 percent of the 
time, compared 
with 89.4 percent 
of the time for 
Black NCVPS 
credit recovery 
students and 
89.5 percent of the 
time for Hispanic 
NCVPS credit 
recovery students 


Table 10. Short- and longer-term outcomes after completing North Carolina Virtual 
Public School credit recovery, by race/ethnicity, 2008/09-2011/12 (percent) 


Outcome 

Asian 

Black 

Hispanic 

White 

Short-term outcome 

Student demonstrates proficiency, all courses combined 

44.4 

25.9 

33.8 

38.7 

Longer-term outcome 

Student re-enrolls following year 

a 

89.4 

89.5 

85.1 

Student passes next course in sequence (all courses 
combined) 

64.9 

65.4 

60.9 

64.6 

Student graduates 

83.7 

78.8 

72.3 

73.5 

If student graduates, graduates on time 
(within four years) 

36.1 

30.6 

37.1 

30.1 


Note: Black includes African American; Hispanic includes Latino. 

a. Percentage omitted because there were fewer than five students in this subgroup for this item. 
Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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One intriguing pattern that may merit additional investigation was among Black NCVPS 
credit recovery students. Even after outcomes were statistically adjusted, Black NCVPS 
credit recovery students were less likely than White NCVPS credit recovery students to 
demonstrate proficiency following credit recovery (as measured by achievement level) but 
more likely to re-enroll, to succeed in subsequent coursework (as measured by earning 
course credit), to graduate, and to graduate on time (see table B4 in appendix B). 

Economic disadvantage status. Patterns for NCVPS credit recovery students who were eco- 
nomically disadvantaged paralleled those for the larger population of students who were not 
economically disadvantaged. For example, NCVPS credit recovery students who were eco- 
nomically disadvantaged had a lower average baseline proficiency rate (28.4 percent) than 
NCVPS credit recovery students who were not economically disadvantaged (35.8 percent) 
and a lower graduation rate after credit recovery (73.1 percent versus 79.2 percent; table 11). 

Even after regression analysis controlled for other possible explanatory factors, these stu- 
dents still appeared to be less likely to graduate and to re-enroll. Their predicted perfor- 
mance typically was lower on subsequent end-of-course proficiency (that is, whether a 
student’s exam score indicates that she or he is performing at grade level) and achievement 
(that is, her or his actual scaled score) measures (though only statistically significantly so 
on the measure of achievement, and even then with only a small effect size). It is again 
worth noting, however, that, as was the case with racial/ethnic minority students, after 
statistical adjustment, students in this sample who were economically disadvantaged were 
more likely to graduate on time than students who were not economically disadvantaged 
(see table B5 in appendix B and tables CIO, C20, C25, and C29-C31 in appendix C). 

Initial rate of failure. Finally, examinations through regression analysis of differences 
in outcomes for students with higher rates of end-of-course exam failure in the first year 
in which they failed one or more courses (that is, students who ended their year farther 
behind academically than their peers the first time they failed one or more high school 
courses) were inconclusive. Differences in shorter-term outcomes were not significant, and 
differences in longer-term outcomes, though almost all statistically significant, were diffi- 
cult to interpret without additional information (see tables CIO, C20, C25, and C29-C31 
in appendix C). Such inconclusive results do not mean that important differences for these 


Table 11. Short- and longer-term outcomes after completing North Carolina Virtual 
Public School credit recovery, by economic disadvantage status, 2008/09-2011/12 

(percent) 






Not 

Difference 


Economically 

economically 

in outcome 

Outcome 

disadvantaged 

disadvantaged 

(percent) 

Short-term outcome 

Student demonstrates proficiency, all courses combined 

28.4 

35.8 

-7.4 

Longer-term outcome 

Student re-enrolls following year 

91.1 

92.3 

-1.2 

Student passes next course in sequence 
(all courses combined) 

65.8 

63.7 

+2.1 

Student graduates 

73.1 

79.2 

-6.1 

If student graduates, graduates on time 
(within four years) 

33.6 

29.3 

+4.3 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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students do not exist when it comes to their participation in online credit recovery, but it 
may require more careful qualitative work to uncover those differences. 

Implications of the study findings 


While there were almost no clear and strong positive associations between participation 
in NCVPS credit recovery and the short- and longer-term academic outcomes examined 
relative to participation in other credit recovery options, there also were few meaning- 
fully negative associations. Also, while the results for different student subgroups within 
the NCVPS credit recovery group are preliminary at best, they do offer possibilities for 
follow-up studies about whether it is beneficial to prioritize participation in online credit 
recovery for certain student populations — especially for Black students. 

North Carolina has demonstrated a persistent and growing interest in continuing to 
expand the use of online and digital instruction statewide. As programs like NCVPS 
continue to grow (NCVPS is now the second-largest online school in the nation, after 
Florida’s; Hughes et al., 2015) and the state continues to emphasize the importance of 
transitioning to digital learning, it will become increasingly important to expand under- 
standing of how to best support that transition. These analyses suggest that at least one 
group of online learners — credit recovery students — differs in potentially important ways 
from credit recovery students in traditional face-to-face programs (for example, in terms of 
academic preparedness and socioeconomic background) and as a result may require sup- 
ports that are different from those provided for traditional students. 

More evidence and more rigorous and controlled studies will be needed to give strength 
to the preliminary findings in this study that there may be a few detriments and, in some 
scenarios (for instance, efforts to keep students on track for on-time graduation), some 
benefits to participation in online credit recovery. This study’s analyses suggest that online 
credit recovery may offer a promising option for a population of students for whom finding 
the right academic experience is critical to their academic success — and often a challenge. 

Limitations of the study 


This study has five main limitations: 

• One of the key advantages of a large, rich, longitudinal dataset like the one used 
for this study’s analyses is that the high number of cases examined helps reduce 
the chances that the analyses have been affected in some way by unusual results 
from a single year or by too few observations to generate useful results. The size 
and richness of the dataset still do not allow drawing conclusions about what 
caused some results revealed by the study’s analyses. But it does help contribute to 
an understanding of where future research should focus to explore the costs and 
benefits of online credit recovery for high school students. 

• The analyses cannot be used to draw any direct conclusions about the quality 
of online credit recovery courses or of their overall efficacy in improving student 
academic outcomes. However, these analyses do provide the kind of information 
necessary to understand how much and what kind of data are available on the 
topic, as well as some of the causal research questions the available data may allow 
future studies to consider. 


The analyses 
suggest that at 
least one group 
of online learners 
-credit recovery 
students-differs 
in potentially 
important ways 
from credit 
recovery students 
in traditional face- 
to-face programs 
(for example, in 
terms of academic 
preparedness and 
socioeconomic 
background) 
and may require 
supports that are 
different from 
those provided 
for traditional 
students 
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• The state funding formula for NCVPS credit recovery changed in the 2011/12 
school year. The study’s analytical design was not affected by this funding change, 
as the credit recovery student sample were all enrolled before the funding change. 
However, participation, and thus the correlations, might change under the new 
funding regime. 

• End'ohcourse exams in North Carolina do not cover all subjects; the tested sub- 
jects have changed over time and the exams themselves have recently changed. 
Anticipating future results based on past testing scenarios is not always practicable. 

• The samples used to conduct the analyses are not necessarily representative of 
all online credit recovery programs or participants. First, most online courses in 
these analyses were NCVPS credit recovery courses; therefore, the results may 
not be applicable to other online settings. Also, the administrative data did not 
explicitly identify credit recovery courses, and the processes used to identify 
those courses could have introduced errors or omissions. All students who par- 
ticipated in third-party online credit recovery programs could not be identified 
(though interactions with third-party vendors did indicate that this population 
is not large). And the decision to pool all non-NCVPS credit recovery courses 
into a single subgroup for comparison to outcomes from NCVPS credit recovery 
courses could have masked important differences in course and student outcomes 
for subpopulations of that larger group (for example, outcomes for summer school 
versus traditional school-year face-to-face credit recovery students versus outcomes 
for students who participated in third-party credit recovery programs). 
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Appendix A. Data and methodology 


This appendix includes detailed information about the data used for this report, as well as 
the methods used to complete the analyses. 

Data 

This section outlines the sources of the data used for this report and details about the 
student data sample. 

Data sources. The data for this study came from three primary sources: North Caroli- 
na Virtual Public School (NCVPS), which provided a description of its credit recovery 
program and student enrollment data that could be matched to state administrative data; 
longitudinal, administrative student records provided by North Carolina Department of 
Public Instruction and hosted and maintained by the Education Research Data Center at 
Duke University; and the seven third-party credit recovery program vendors in operation 
in the state, who provided descriptions of their programs. Information for the description 
of third-party credit recovery options was gathered through telephone interviews with rep- 
resentatives of those vendors (see appendix D). 

Administrative records provided student-level data and included many individual demo- 
graphic variables — for example, age, race, economic disadvantage status (as proxied by 
eligibility for the federal school lunch program) — and academic variables — for example, 
performance on standardized state end-of-course exams, course enrollment. 

Sample. The sample included all identifiable students who took credit recovery courses 
offered by NCVPS between 2008/09 — the first full academic year of NCVPS’s program — 
and 2011/12, as well as all identifiable students whose credit recovery involved re-enrolling 
in a course during the regular school year (traditional face-to-face credit recovery), during 
summer school, or, when possible, as part of some other credit recovery program, such 
as participation in a program provided by a third-party vendor (see box A1 for efforts to 
identify participants in other credit recovery options). For some research questions, student 
records from the preceding year (2007/08) were included to provide baseline (pre-participa- 
tion) measures for some students. 

Because students can fail more than one course and can, therefore, choose more than 
one type of credit recovery, for most analyses data were organized by instances of course 
enrollment instead of by student. Thus, a single student could have separate records in the 
data for credit recovery courses taken online or face to face, as well as separate records for 
courses taken in different years. The sample of all credit recovery students included about 

14.900 instances of NCVPS credit recovery enrollment between fall 2008 and spring 2012, 
about 329,900 instances of face-to-face traditional credit recovery enrollment, and about 

33.900 instances of summer school credit recovery enrollment (see tables 1 through 3 in 
the main text). Courses included in the analyses were limited to the 12 offered by NCVPS 
as part of its credit recovery program — English I through IV, Algebra I and II, Geometry, 
Physical Science, Biology, Civics and Economics, U.S. History, and World History. 
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Box Al. Participation in third-party and other credit recovery options 

As noted in the main text, several North Carolina school districts have employed third-party 
credit recovery services in addition to North Carolina Virtual Public School (NCVPS) credit 
recovery and traditional, school-year and summer school course repetition. The first step in 
collecting information about these programs was to contact each third-party vendor believed to 
be in operation in North Carolina between 2008/09 and 2011/12, based on initial information 
provided by NCVPS. In most cases the initial point of contact was a vendor’s marketing office. 
The next step was to contact regional representatives to learn more about the company and its 
products. Following each conversation, a vendor representative validated meeting notes and 
supplemented information that was not addressed during the interview. 

In the end, information derived from contact with these vendors was incomplete, in most 
cases limited to overall enrollment numbers or overall numbers of subscribing districts. Also, it 
became apparent during preparation of the data for this report that the responsibility for devel- 
oping procedures for recording transcript data falls to each school district (there were 115 
traditional school districts and, during the time period covered by this study, up to 100 charter 
schools), resulting in inconsistencies across student records in the ways in which credit recov- 
ery courses are identified. In particular, some districts do not include credit recovery courses 
from third-party vendors on a formal transcript. As a result, it was not possible to determine 
accurate estimates of third-party credit recovery program participants. 

However, it was possible to identify a group of students who experienced some form of 
otherwise unaccounted-for credit recovery (for example, students who failed a required course 
in a given sequence without evidence of subsequent credit recovery, but who were on record 
as taking the next course in the sequence). These students were identified as “other credit 
recovery’’ students in the data and likely include some of the third-party credit recovery 
population. For the analyses described below, this subset of course instances (only about 
2.2 percent of the total number of course instances — about 7,700 in all) were included as 
part of the non-NCVPS credit recovery records (because they were clearly identifiable as non- 
NCVPS credit recovery courses) but were not broken out separately from the other non-NCVPS 
credit recovery courses (because variations in course-recording procedures across districts 
do not allow for definitive conclusions about what type of non-NCVPS credit recovery these 
students accessed). 


The sample used for analyses of differences among NCVPS credit recovery students includ- 
ed only the 12,900 students for whom complete administrative data could be matched to 
evidence of participation in NCVPS credit recovery. 

In some cases a subset of students was excluded for some research questions, because 
they had not been enrolled long enough to experience some of the mid- or longer-term 
outcomes of interest. In other cases students were excluded if their records could not be 
matched to demographic information required for analysis. However, rates of matching 
students to the demographic and transcript data necessary for many of the analyses were 
very high (for example, 93 percent of all NCVPS credit recovery students were identifiable 
in the official state administrative dataset). 
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Quantitative analysis methodology 


This section outlines the strategies used to determine categorization of each record of 
enrollment in a credit recovery course, and details about the regression analyses conducted 
after completion of this categorization. 

Identifying NCVPS online credit recovery courses. Because the North Carolina Depart- 
ment of Public Instruction administrative database does not contain an online course 
identifier, and because (as noted in box Al) individual districts are responsible for encod- 
ing student enrollment, the process of identifying the various instances of credit recovery 
required several steps. The first step was to match NCVPS’s records to the state records, 
resulting in a very high match rate (93 percent). 

Identifying traditional school-year and summer school credit recovery courses. Next, 
records of all other courses for the years of interest were selected, then reduced to include 
only those that matched the 12 NCVPS credit recovery courses. Despite differences in 
the course titles and in course coding schemes across districts, all districts do include a 
universal initial four-digit course code that identifies the discipline (first two numbers) and 
the course (second two numbers). By identifying instances of targeted courses that were 
repeated (and for which initial enrollment included an indication of no credit earned), 
the study team could deduce (from information about the semester in which the repeated 
courses were taken) whether those courses were traditional repetitions (that is, taken in 
the fall or spring semester) or summer school repetitions. Because credit recovery requires 
repeating courses and because the transcript data begin in 2007/08, the credit recovery 
course analyses are limited to 2008/09-2011/12. 

Overview of quantitative analyses. All research questions were intended either to provide 
descriptive information or to generate correlations related to participation in various credit 
recovery programs. Because this study was not experimental or quasi-experimental, causal 
inferences could not be made. For all components of research questions 2 and 3, the study 
team used multivariate ordinary least squares regression models for continuous outcomes 
(for example, scaled exam scores) and multivariate logistic regression models for binary 
outcomes (for example, whether a student graduated). Definitions of the outcome variables 
are included in box A2, more details explanations of the regression analyses are included 
in box A3, and examples of these models are included in each analysis section below. 

This study is both exploratory and preliminary in nature in that it is a first attempt to 
understand better the place of NCVPS credit recovery in the landscape of credit recovery 
in North Carolina. It is not an attempt to draw definitive conclusions about outcomes 
for various specific credit recovery pathways. Therefore, all regression analyses compared 
two populations: students who took a NCVPS credit recovery course and credit recovery 
students who did not. Limiting the analyses for research questions 2 and 3 to comparisons 
and contrasts for these two groups (rather than for all four groups — NCVPS, tradition- 
al school-year, summer school, and third-party credit recovery — separately) significantly 
reduced reporting complexity and also reduced the possibility that study results might be 
misinterpreted as identifying causal impacts of one specific credit recovery option over 
another. 
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Box A2. Definitions of outcome variables 

Achievement levels and scaled scores. Outcomes on North Carolina end-of-grade (elementary 
and middle school) and end-of-course (high school) exams for the years included in this study 
are measured in two ways: whether those outcomes indicate that a student has reached a 
certain level of proficiency (four-point scale, with a 3 or 4 indicating proficiency); and a scaled 
score result, the scale of which is common across exam subjects. For cases in which students 
retook an exam in the same testing period (for example, before student enrollment in or com- 
pletion of credit recovery), the study team followed the state’s procedure of using the higher of 
the two exam scores as the official first score of record. 

Re-enrollment. A student was identified as having re-enrolled if there was evidence that she or 
he enrolled in any courses in the year following the year of interest and that she or he had not 
officially graduated the previous year. 

Success in subsequent courses. A student’s success in a later course was determined by 
identifying students in the sample who were enrolled in and earned credit for a credit recovery 
course of interest and later enrolled in the next course in the subject sequence. Because some 
science and history courses can be taken out of sequence, only math and English courses 
were included in these analyses. 

Graduation. A student was identified as having graduated based solely on the presence of a 
graduation flag in the administrative data. 

On-time graduation A student was identified as having graduated on time if her or his first 
year of enrollment in grade 9 could be identified, and her or his graduation year was within four 
years of that first matriculation year. 


Box A3. Regression methods 

Regression analysis for outcomes with many potential values. One type of regression analy- 
sis used in this study is ordinary least squares regression, which is used when the outcome 
of interest has many possible values — such as an exam score. For example, students in one 
school district might appear to be less likely to graduate on time than students in other dis- 
tricts based on their graduation rates alone, but when other aspects of their schooling that 
may have influenced their chances for graduation (such as family income level) are factored 
out mathematically, it may turn out that going to school in a certain district did not really affect 
graduation rates at all. 

Regression analysis for outcome variables with only two potential values. Another type of 
regression analysis, logistic regression, was used when the outcome of interest had only 
two possible values — such as yes or no. This type of analysis generates predicted odds for 
whether the outcome occurred for students with certain characteristics, all else being equal. 
For example, if the outcome of interest is whether or not a student graduates on time, this kind 
of regression can generate the adjusted odds that different types of students (for example, 
English learner students, racial/ethnic minority students, students who live in rural communi- 
ties) experienced that outcome by factoring out the influence of other characteristics. 
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Given that credit recovery takes place within definable clusters — such as individual dis- 
tricts and specific implementation years — the study team extended the analyses to investi- 
gate whether multilevel modeling options were required, but did not find evidence for the 
need for multilevel modeling techniques (for example, intraclass correlations were very 
low at <0.03 in most cases, as was the amount of variance, typically <1 percent, explained 
by three non-student-level grouping variables: grade, district, and year). Consequently, all 
results are reported for singledevel ordinary least squares and logistic regressions. 

Achievement outcomes and outcomes related to success in subsequent courses were ana- 
lyzed independently by course, and also pooled across courses to increase statistical power. 
Because of the variety of course combinations students could take across their high school 
careers, other academic outcomes (such as graduation on time) were analyzed across all 
courses and not by specific credit recovery course. 

Analyses of short-term outcomes. The student population for these analyses included all 
students who retook a state standardized end-oficourse exam after completion of credit 
recovery. The population was limited by two factors. First, students who failed a course but 
initially passed the mandatory end-oficourse exam did not have to retake the exam after 
completion of the credit recovery course. Second, end-oficourse exams were limited to 
only a subset of the courses offered for credit recovery by NCVPS, and in some cases were 
limited to a subset of years (table Al). 

The study team used a multivariate ordinary least squares regression to predict students’ 
course-specific normalized end-oficourse exam scores following NCVPS credit recovery 
participation. Also, multivariate logistic regressions predict one related binary dependent 
variable following students’ participation in NCVPS credit recovery: whether the student 
subsequently reached proficiency in the same course’s end-oficourse exam. During the 
years of interest for this study, proficiency was designated by a 3 or 4 (on a four-point scale). 

Because the scale and achievement level designations are the same across exams, these two 
dependent variables were modeled not only independently by course but also in aggregate. 


Table Al. North Carolina Virtual Public School credit recovery courses with end-of- 


course exams, 2008/09-2011/12 

1 Course 

2008/09 

2009/10 

2010/11 

2011/12 

English 1 

✓ 

✓ 

✓ 

✓ 

Algebra 1 

✓ 

✓ 

✓ 

✓ 

Algebra II 

✓ 

✓ 

✓ 


Geometry 

✓ 

✓ 



Physical Science 

✓ 

✓ 

✓ 


Biology 

a 

✓ 

✓ 

✓ 

Civics and Economics 

✓ 

✓ 

✓ 


U.S. History 

✓ 

✓ 

✓ 



a. North Carolina administered Biology end-of-course exams in 2008/09, but North Carolina Virtual Public 
School did not offer a Biology credit recovery option that year. 

Source: North Carolina Department of Public Instruction, n.d. a. 




Short-term outcomes model example 


End-of-Course Scaled Score: English I Retest = 

(3 0 + (Jj * Enroll in NCVPS Credit Recovery + 

(3 2 * Initial English I End-of-Course Scaled Score + j3 3 * English Learner + 

(3 4 * Economically Disadvantaged + (3 5 * Black + (3 6 * Hispanic + j3 ? * Asian + 

(3 g * American Indian/Alaska Native + (3 9 * Hawaiian Native/Pacific Islander + 

(3 10 * Multiracial + (3 n * Special Education Status + (3 12 * Rural + (3 13 * Female + 

(3 14 * Grade in School + (3 15 * Days in Attendance This School Year + 

(3 16 * School Year + error. 

Student Demonstrates Proficiency on End-of-Course English I Retest = 

^ Student Demonstrates Proficiency 

1 - Student Demonstrates Proficiency 0 
(3j * Enroll in NCVPS Credit Recovery + (3 2 * Initial English I End-of-Course Scaled Score + 
(3 3 * English Learner + j3 4 * Economically Disadvantaged + j3 5 * Black + j3 6 * Hispanic + 
j3 ? * Asian + (3 g * American Indian/Alaska Native + (3 9 * Hawaiian Native/Pacific Islander + 
(3 10 * Multiracial + (3 n * Special Education Status + (3 12 * Rural + ^ * Female + 

(3 14 * Grade in School + [3 15 * Days in Attendance This School Year + 

(3 16 * School Year + error. 

Analyses of mid- and longer-term outcomes. The population for these analyses included 
all students for whom dropout data or re'enrollment data were available after taking a 
credit recovery course. As with the previous research question, this population was limited 
by two factors. 

First, because data for this study extend only through the 2011/12 school year, the analyses 
of multiyear outcomes such as eventual graduation were limited to those students for whom 
the study team had the longitudinal records necessary to answer each part of this research 
question. For example, a student who re-took an English I course during the 2011/12 school 
year does not have matching data about her or his success in 2012/13; similarly, a student 
who enters the dataset during grade 9 in the 2010/11 school year does not have subsequent 
graduation data. 

Second, analysis of success in subsequent courses in a sequence is limited by the number of 
courses that are mandatory for high school graduation in North Carolina. For example, stu- 
dents who take Algebra I may complete their math requirements for graduation with that 
course, and therefore have no subsequent math course data. Analyses related to success in 
subsequent courses were limited to the Algebra I -► Algebra II/Geometry sequence and the 
English I -> English II ->■ English III ->■ English IV sequence. 

All analyses for the components of this research question employed multivariate logistic 
regressions. 
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Mid- to longer-term outcomes model example 


Student Graduates = 

L Student Graduates 
1 - Student Graduates 


= (3 0 + * Enroll in NC VPS Credit Recovery 


(i 2 * English Learner + (J 3 * Economically Disadvantaged + j^ 4 * Black + j^ 5 * Hispanic + 

(i 6 * Asian + * American Indian/Alaska Native + (i 8 * Hawaiian Native/Pacific Islander + 


(i 9 * Multiracial + ^ * Special Education Status + (},,* Rural 


P 13 * Grade in School 


n n».» ■ (i 12 * Female 
(i 14 * Days in Attendance This School Year + 


(J 15 * School Year + error. 


Student Succeeds in English II after Completion of English I Credit Recovery = 

Student Graduates 


Log 


= (3 0 + * Enroll in NCVPS Credit Recovery 


1 - Student Graduates 

(i 2 * English Learner + * English I End-of-Course Retake Scaled Score + 

* Economically Disadvantaged + (J 5 * Black + (J 6 * Hispanic + j^ 7 * Asian + 
ji g * American Indian/Alaska Native + (i 9 * Hawaiian Native/Pacific Islander + 
ji 10 * Multiracial + jj * Special Education Status + (i 12 * Rural + ji 13 * Female + 


(i 14 * Grade in School + (i 15 * Days in Attendance This School Year 
(i 16 * School Year + error. 


A-7 


Appendix B. Regression results referenced in the main text 


This appendix includes results of selected regression analyses referenced in the main report. 
The raw analyses from which these tables were generated can be found in appendix C. 


Table Bl. Predicted success on North Carolina end-of-course exams after 
completing credit recovery, 2008/09-2011/12 




Odds ratio results 

Course 

Likelihood NCVPS credit recovery students demonstrated 
proficiency relative to other credit recovery students 3 

English 1 


Less likely 

Algebra 1 


Less likely 

Algebra II 


Less likely 

Geometry 


Less likely 

Physical Science 


Less likely 

Biology 


Less likely 

Civics and Economics 


Less likely 

U.S. History 


Less likely 

All courses combined 


Less likely 


Predicted change in scaled scores 


NCVPS credit recovery students relative Effect size 

to other credit recovery students 3 for predicted change 

English 1 

-0.7 

0.0005 

Algebra 1 

-1.6 

0.0015 

Algebra II 

-2.1 

0.0042 

Geometry 

-2.5 

0.0048 

Physical Science 

-2.3 

0.0036 

Biology 

-2.3 

0.0027 

Civics and Economics 

-1.6 

0.0024 

U.S. History 

-1.2 

0.0009 

All courses combined 

-1.6 

0.0018 


NCVPS is North Carolina Virtual Public School. 

a. All results are statistically significant. See appendix A for methodology details and appendix C for expanded 
results. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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Table B2. Predicted longer-term in-school outcomes after completing credit 
recovery in North Carolina, 2008/09-2011/12 

Statistically adjusted results 

Outcome 

Likelihood NCVPS credit recovery students 
achieved longer term outcomes relative 
to other credit recovery students 

Student re-enrolls following year 

More likely 3 

Student passes next course in sequence: 

English II after English 1 credit recovery 

Less likely 

English III after English II credit recovery 

Less likely 

English IV after English III credit recovery 

Less likely 

Algebra II or Geometry, after Algebra 1 credit recovery 

Less likely 


NCVPS is North Carolina Virtual Public School. 

a. Result is statistically significant. See appendix A for methodology details and appendix C for expanded 
results. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 


Table B3. Predicted longer-term graduation outcomes after completing credit 
recovery in North Carolina, 2008/09-2011/12 



Statistically adjusted results 

Outcome 

Likelihood NCVPS credit recovery students 
graduated relative to other credit recovery students 

Student graduates 

Less likely 3 

If student graduates, graduates on time (within four years) 

More likely 3 


NCVPS is North Carolina Virtual Public School. 

a. Result is statistically significant. See appendix A for methodology details and appendix C for expanded 
results. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 


Table B4. Predicted short- and longer-term outcomes for Black students 
after completing North Carolina Virtual Public School credit recovery, 
2008/09-2011/12 


Outcome 

Statistically adjusted results 

Likelihood Black NCVPS credit recovery 
students achieved outcome relative to 
White NCVPS credit recovery students 3 

Short-term outcome 

Student demonstrates proficiency, all courses combined 

Less likely 

Longer-term outcome 

Student re-enrolls following year 

More likely 

Student passes next course in sequence (all courses combined) 

More likely 

Student graduates 

More likely 

If student graduates, graduates on time (within four years) 

More likely 


NCVPS is North Carolina Virtual Public School. 

a. All results are statistically significant. See appendix A for methodology details and appendix C for expanded 
results. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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Table B5. Predicted short- and longer-term outcomes after completing North 
Carolina Virtual Public School credit recovery, by economic disadvantage status, 
2008/09-2011/12 

Outcome 

Predicted change in 
scaled scores 

NCVPS credit recovery students 
who were economically 
disadvantaged relative to other Effect 

NCVPS credit recovery students size 

Short-term continuous outcome 

All courses combined 

-0.42 a 0.0014 

Outcome 

Statistically adjusted results 
Likelihood NCVPS credit recovery 
students who were economically 
disadvantaged achieved outcome relative 
to other NCVPS credit recovery students 

Short-term binary outcome 

Student demonstrates proficiency, all courses combined 

Less likely 

Longer-term binary outcome 

Student re-enrolls following year 

Less likely 3 

Student passes next course in sequence (all courses combined) 

No difference 

Student graduates 

Less likely 3 

If student graduates, graduates on time (within four years) 

More likely 3 


NCVPS is North Carolina Virtual Public School. 

a. Result is statistically significant. See appendix A for methodology details and appendix C for expanded 
results. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 


B-3 







Appendix C. Detailed results of all regression analyses 


This appendix includes tables of the raw results from all ordinary least squares regression 
analyses (analyses for which the outcome of interest can have several different values) and 
logistic regression analyses (analyses for which the outcome of interest has only two pos- 
sible values) completed for this report, as described in appendix A. The general outcome 
tables in this appendix include the data used to construct the tables in appendix B. 

Results of analyses of short-term outcomes 


Table Cl. Demonstration of proficiency on North Carolina end-of-course exam 
retest, all courses combined, 2008/09-2011/12 


1 Variable 

Odds ratio 

Standard error 

Significance p 

NCVPS credit recovery student 

0.6119 

0.0273 

*** 

First exam scaled score 

1.1920 

0.0017 

*** 

English learner student 

0.4920 

0.0170 

*** 

Economically disadvantaged 

0.8822 

0.0125 

*** 

Black 

0.5568 

0.0086 

*** 

Hispanic 

0.9944 

0.0280 


Asian 

0.9513 

0.0634 


American Indian/Alaskan Native 

0.6382 

0.0433 

*** 

Hawaiian Native/Pacific Islander 

0.9499 

0.4351 


Multiracial 

0.8796 

0.0363 

** 

Students receiving special education services 

0.5455 

0.0097 

*** 

Rural 

0.9559 

0.0130 

*** 

Female 

0.9013 

0.0120 

*** 

Grade 

1.0912 

0.0068 

*** 

Attendance 

1.0058 

0.0003 

*** 

Year 

0.9945 

0.0060 



* significant at p < .05; ** significant at p < .01, *** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Number of course records = 125,560; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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Table C2. Demonstration of proficiency on North Carolina end-of-course exam 
retest, English 1, 2008/09-2011/12 

1 Variable 

Odds ratio 

Standard error 

Significance I 

NCVPS credit recovery English 1 student 

0.8452 

0.0892 


First English 1 exam scaled score 

1.2268 

0.0047 

*** 

English learner student 

0.3402 

0.0287 

*** 

Economically disadvantaged 

0.7986 

0.0303 

*** 

Black 

0.5228 

0.0208 

*** 

Hispanic 

1.0570 

0.0781 


Asian 

0.6444 

0.1362 

* 

American Indian/Alaskan Native 

0.4845 

0.0764 

*** 

Hawaiian Native/Pacific Islander 

2.3714 

2.2086 


Multiracial 

0.9118 

0.1011 


Students receiving special education services 

0.4860 

0.0194 

*** 

Rural 

0.8793 

0.0307 

*** 

Female 

0.9629 

0.0343 


Grade 

1.0231 

0.0301 


Attendance 

1.0037 

0.0007 

*** 

Year 

0.9590 

0.0138 

** 


* significant at p < .05, ** significant at p < .01, *** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Number of course records = 21,959; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 


Table C3. Demonstration of proficiency on North Carolina end-of-course exam 
retest, Algebra 1, 2008/09-2011/12 

1 Variable 

Odds ratio 

Standard error 

Significance 

NCVPS credit recovery Algebra 1 student 

0.6493 

0.0554 

*** 

First Algebra 1 exam scaled score 

1.1904 

0.0031 

*** 

English learner student 

0.5633 

0.0351 

*** 

Economically disadvantaged 

0.9119 

0.0235 

*** 

Black 

0.5328 

0.0149 

*** 

Hispanic 

0.9360 

0.0473 


Asian 

0.8948 

0.1162 


American Indian/Alaskan Native 

0.6394 

0.0772 

*** 

Hawaiian Native/Pacific Islander 

0.5283 

0.4615 


Multiracial 

0.8164 

0.0574 

** 

Students receiving special education services 

0.5566 

0.0177 

*** 

Rural 

1.0041 

0.0247 


Female 

0.9627 

0.0232 


Grade 

0.9127 

0.0130 

*** 

Attendance 

1.0074 

0.0006 

*** 

Year 

1.0200 

0.0102 



** significant at p < .01, *** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Number of course records = 39,355; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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Table C4. Demonstration of proficiency on North Carolina end-of-course exam 
retest, Algebra II, 2008/09-2010/11 

1 Variable 

Odds ratio 

Standard error 

Significance j 

NCVPS credit recovery Algebra II student 

0.5781 

0.0766 

*** 

First Algebra II exam scaled score 

1.1358 

0.0060 

*** 

English learner student 

0.7671 

0.1206 


Economically disadvantaged 

0.9071 

0.0478 


Black 

0.6497 

0.0366 

*** 

Hispanic 

0.9530 

0.0953 


Asian 

1.0341 

0.1881 


American Indian/Alaskan Native 

0.9393 

0.2642 


Hawaiian Native/Pacific Islander 

1.0000 

(omitted) 


Multiracial 

0.8565 

0.1212 


Students receiving special education services 

0.6565 

0.0703 

*** 

Rural 

1.1662 

0.0577 

** 

Female 

1.0672 

0.0519 


Grade 

0.9379 

0.0353 


Attendance 

1.0083 

0.0014 

*** 

Year 

1.1300 

0.0326 

*** 


** significant at p < .01, *** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Number of course records = 7,923; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 


Table C5. Demonstration of proficiency on North Carolina end-of-course exam 
retest, Geometry, 2008/09-2009/10 

1 Variable 

Odds ratio 

Standard error 

Significance 

NCVPS credit recovery Geometry student 

0.3750 

0.0765 

*** 

First Geometry exam scaled score 

1.1957 

0.0081 

*** 

English learner student 

0.8332 

0.1527 


Economically disadvantaged 

0.9382 

0.0604 


Black 

0.5781 

0.0422 

*** 

Hispanic 

0.7705 

0.0988 

* 

Asian 

0.8333 

0.2183 


American Indian/Alaskan Native 

0.6738 

0.2473 


Hawaiian Native/Pacific Islander 

1.0000 

(omitted) 


Multiracial 

0.7005 

0.1231 

* 

Students receiving special education services 

0.5906 

0.0740 

*** 

Rural 

0.9905 

0.0610 


Female 

0.9119 

0.0553 


Grade 

0.8128 

0.0356 

*** 

Attendance 

1.0097 

0.0019 

*** 

Year 

1.3700 

0.0720 

*** 


* significant at p < .05, *** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Number of course records = 5,579; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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Table C6. Demonstration of proficiency on North Carolina end-of-course exam 
retest, Physical Science, 2008/09-2010/11 

1 Variable 

Odds ratio 

Standard error 

Significance j 

NCVPS credit recovery Physical Science student 

0.5065 

0.0868 

*** 

First Physical Science exam scaled score 

1.1684 

0.0062 

*** 

English learner student 

0.6928 

0.1089 

** 

Economically disadvantaged 

0.9987 

0.0581 


Black 

0.5646 

0.0356 

*** 

Hispanic 

1.0941 

0.1422 


Asian 

1.0670 

0.3031 


American Indian/Alaskan Native 

0.9433 

0.3249 


Hawaiian Native/Pacific Islander 

1.0000 

(omitted) 


Multiracial 

1.2004 

0.2495 


Students receiving special education services 

0.5637 

0.0412 

*** 

Rural 

0.9508 

0.0558 


Female 

1.0235 

0.0567 


Grade 

1.3040 

0.0470 

*** 

Attendance 

1.0058 

0.0011 

*** 

Year 

0.9720 

0.0318 



** significant at p < .01, *** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Number of course records = 7,002; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 


Table C7. Demonstration of proficiency on North Carolina end-of-course exam 
retest, Biology, 2009/10-2011/12 

1 Variable 

Odds ratio 

Standard error 

Significance jj 

NCVPS credit recovery Biology student 

0.4432 

0.0615 

*** 

First Biology exam scaled score 

1.1830 

0.0044 

*** 

English learner student 

0.4868 

0.0409 

*** 

Economically disadvantaged 

0.9196 

0.0331 

** 

Black 

0.5579 

0.0225 

*** 

Hispanic 

1.0224 

0.0707 


Asian 

1.0702 

0.1916 


American Indian/Alaskan Native 

0.5078 

0.0951 

*** 

Hawaiian Native/Pacific Islander 

0.9995 

0.9303 


Multiracial 

0.8562 

0.0912 


Students receiving special education services 

0.6087 

0.0269 

*** 

Rural 

1.0051 

0.0341 


Female 

0.8934 

0.0297 

*** 

Grade 

1.1594 

0.0242 

*** 

Attendance 

1.0046 

0.0008 

*** 

Year 

1.0075 

0.0150 



** significant at p < .01, *** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Number of course records = 18,983; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. North Carolina administered Biology end-of- 
course exams in 2008/09, but NCVPS did not offer a Biology credit recovery option that year. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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Table C8. Demonstration of proficiency on North Carolina end-of-course exam 
retest, Civics and Economics, 2008/09-2010/11 

1 Variable 

Odds ratio 

Standard error 

Significance 1 

NCVPS credit recovery Civics and Economics student 

0.5594 

0.0682 

*** 

First Civics and Economics exam scaled score 

1.2280 

0.0056 

*** 

English learner student 

0.5140 

0.0494 

*** 

Economically disadvantaged 

0.9271 

0.0377 


Black 

0.5071 

0.0231 

** * 

Hispanic 

1.0067 

0.0710 


Asian 

0.7747 

0.1368 


American Indian/Alaskan Native 

0.7878 

0.1398 


Hawaiian Native/Pacific Islander 

0.2671 

0.3674 


Multiracial 

0.9632 

0.1180 


Students receiving special education services 

0.6452 

0.0325 

*** 

Rural 

0.8512 

0.0335 

*** 

Female 

0.7182 

0.0272 

*** 

Grade 

1.1803 

0.0298 

** * 

Attendance 

1.0051 

0.0010 

* * * 

Year 

0.9249 

0.0197 

* * * 


*** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Number of course records = 16,152; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 


Table C9. Demonstration of proficiency on North Carolina end-of-course exam 
retest, U.S. History, 2008/09-2010/11 


1 Variable 

Odds ratio 

Standard error 

Significance 

NCVPS credit recovery U.S. History student 

0.6270 

0.1121 

** 

First U.S. History exam scaled score 

1.2148 

0.0069 

*** 

English learner student 

0.6391 

0.0996 

** 

Economically disadvantaged 

0.9121 

0.0485 


Black 

0.7587 

0.0442 

*** 

Hispanic 

1.0180 

0.1221 


Asian 

1.5476 

0.3958 


American Indian/Alaskan Native 

1.0146 

0.2905 


Hawaiian Native/Pacific Islander 

1.0000 

(omitted) 


Multiracial 

0.9106 

0.1596 


Students receiving special education services 

0.6588 

0.0485 

*** 

Rural 

0.9541 

0.0498 


Female 

0.7030 

0.0355 

*** 

Grade 

1.2519 

0.0650 

*** 

Attendance 

1.0055 

0.0011 

*** 

Year 

1.0828 

0.0309 

** 

** significant at p < .01, *** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Number of course records = 8,603; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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Table CIO. Demonstration of proficiency on North Carolina end-of-course exam retest, 
all courses combined, North Carolina Virtual Public School only, 2008/09-2011/12 


1 Variable 

Odds ratio 

Standard error 

Significance 

First exam scaled score 

1.1670 

0.0134 

*** 

English learner student 

0.4157 

0.1256 

** 

Economically disadvantaged 

0.9312 

0.0968 


Black 

0.7891 

0.0922 

* 

Hispanic 

1.2480 

0.2661 


Asian 

1.4619 

0.9587 


American Indian/Alaskan Native 

1.2210 

0.7705 


Hawaiian Native/Pacific Islander 

1.0000 

(omitted) 


Multiracial 

0.6726 

0.1665 


Students receiving special education services 

0.6855 

0.0897 

** 

Rural 

0.8765 

0.0934 


Female 

1.0328 

0.1035 


Grade 

0.8593 

0.0407 

*** 

Attendance 

1.0089 

0.0027 

*** 

Number of end-of-course exams failed first year 

0.8893 

0.0662 


Number of end-of-course exams failed this year 

0.3230 

0.0205 

*** 

Year 

0.8003 

0.0415 

*** 


* significant at p < .05, ** significant at p < .01, *** significant at p < .001. 

Note: Number of course records = 2,924; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 


Table Cll. Exam score retest coefficients for North Carolina end-of-course exam, 
all courses combined, 2008/09-2011/12 

Variable 

Coefficient 

Standard 

error 

Significance 

Effect 

size 

NCVPS credit recovery student 

-1.5813 

0.1084 

*** 

0.0017 

First exam scaled score 

-0.4149 

0.0031 

*** 

0.1277 

English learner student 

-2.3696 

0.0846 

*** 

0.0064 

Economically disadvantaged 

-0.4153 

0.0363 

*** 

0.0011 

Black 

-1.7502 

0.0399 

*** 

0.0155 

Hispanic 

-0.0381 

0.0726 


0.0000 

Asian 

-0.0789 

0.1646 


0.0000 

American Indian/Alaskan Native 

-1.0875 

0.1720 

*** 

0.0003 

Hawaiian Native/Pacific Islander 

0.5978 

1.1259 


0.0000 

Multiracial 

-0.4033 

0.1080 

*** 

0.0001 

Students receiving special education services 

-1.9382 

0.0421 

*** 

0.0171 

Rural 

-0.1181 

0.0342 

*** 

0.0001 

Female 

-0.0721 

0.0338 

* 

0.0000 

Grade 

0.5536 

0.0157 

*** 

0.0101 

Attendance 

0.0184 

0.0008 

*** 

0.0047 

Year 

-0.2469 

0.0153 

*** 

0.0021 


* significant at p < .05, *** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Number of course records = 121,802; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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Table C12. Exam score retest coefficients for North Carolina end-of-course exam, 


English 1, 2008/09-2011/12 

Variable 

Coefficient 

Standard 

error 

Significance 

Effect 

size 

NCVPS credit recovery English 1 student 

-0.6914 

0.2240 

** 

0.0005 

First English 1 exam scaled score 

0.5632 

0.0066 

*** 

0.2547 

English learner student 

-2.8471 

0.1747 

*** 

0.0124 

Economically disadvantaged 

-0.3885 

0.0831 

*** 

0.0010 

Black 

-1.6734 

0.0873 

*** 

0.0171 

Hispanic 

-0.0337 

0.1622 


0.0000 

Asian 

-1.2038 

0.4010 

** 

0.0004 

American Indian/Alaskan Native 

-1.7592 

0.3296 

*** 

0.0014 

Hawaiian Native/Pacific Islander 

2.1655 

2.3198 


0.0000 

Multiracial 

-0.4628 

0.2524 


0.0002 

Students receiving special education services 

-2.1234 

0.0809 

*** 

0.0316 

Rural 

-0.3657 

0.0745 

*** 

0.0011 

Female 

0.2488 

0.0773 

*** 

0.0005 

Grade 

0.0905 

0.0626 


0.0001 

Attendance 

0.0124 

0.0015 

*** 

0.0033 

Year 

-0.2179 

0.0312 

*** 

0.0023 

** significant at p < .01, *** significant at p < 

NCVPS is North Carolina Virtual Public School. 

.001. 





Note: Number of course records = 21,095; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. 

Source: Authors’ analysis of administrative data provided by the Education Research Data Center. 


Table C13. Exam score retest coefficients for North Carolina end-of-course exam, 
Algebra 1, 2008/09-2011/12 

Variable 

Coefficient 

Standard 

error 

Significance 

Effect 

size 

NCVPS credit recovery Algebra 1 student 

-1.5675 

0.2094 

*** 

0.0015 

First Algebra 1 exam scaled score 

0.5945 

0.0057 

*** 

0.2207 

English learner student 

-1.9409 

0.1575 

*** 

0.0039 

Economically disadvantaged 

-0.4483 

0.0670 

*** 

0.0012 

Black 

-1.8629 

0.0735 

*** 

0.0164 

Hispanic 

-0.1579 

0.1344 


0.0000 

Asian 

-0.0343 

0.3271 


0.0000 

American Indian/Alaskan Native 

-0.7375 

0.3101 

* 

0.0001 

Hawaiian Native/Pacific Islander 

1.4281 

2.1019 


0.0000 

Multiracial 

-0.3806 

0.1880 

* 

0.0001 

Students receiving special education services 

-1.9267 

0.0760 

*** 

0.0164 

Rural 

-0.0781 

0.0631 


0.0000 

Female 

0.1356 

0.0622 

* 

0.0001 

Grade 

-0.1638 

0.0360 

*** 

0.0005 

Attendance 

0.0228 

0.0015 

*** 

0.0061 

Year 

-0.1696 

0.0260 

*** 

0.0011 


* significant at p < .05, *** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Number of course records = 38,435; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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Table C14. Exam score retest coefficients for North Carolina end-of-course exam, 
Algebra II, 2008/09-2010/11 

Variable 

Coefficient 

Standard 

error 

Significance 

Effect 

size 

NCVPS credit recovery Algebra II student 

-2.1185 

0.3736 

*** 

0.0042 

First Algebra II exam scaled score 

0.4699 

0.0134 

*** 

0.1375 

English learner student 

-0.5639 

0.4484 


0.0002 

Economically disadvantaged 

-0.2699 

0.1477 


0.0004 

Black 

-1.7126 

0.1584 

*** 

0.0149 

Hispanic 

0.0087 

0.2798 


0.0000 

Asian 

0.6118 

0.4987 


0.0002 

American Indian/Alaskan Native 

-1.1113 

0.7615 


0.0003 

Hawaiian Native/Pacific Islander 

6.7147 

4.1671 


0.0003 

Multiracial 

-0.7203 

0.3956 


0.0004 

Students receiving special education services 

-1.3584 

0.2997 

*** 

0.0027 

Rural 

0.5888 

0.1380 

*** 

0.0024 

Female 

0.2037 

0.1360 


0.0003 

Grade 

-0.3297 

0.1040 

** 

0.0013 

Attendance 

0.0254 

0.0037 

* * * 

0.0060 

Year 

0.1357 

0.0806 


0.0004 


** significant at p < .01, *** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Number of course records = 7,723; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. 

Source: Authors’ analysis of administrative data provided by the Education Research Data Center. 


Table C15. Exam score retest coefficients for North Carolina end-of-course exam, 
Geometry, 2008/09-2009/10 


Variable 

Coefficient 

Standard 

error 

Significance 

Effect 

size 

NCVPS credit recovery Geometry student 

-2.5143 

0.4906 

*** 

0.0048 

First Geometry exam scaled score 

0.5531 

0.0147 

*** 

0.2061 

English learner student 

-1.0904 

0.4674 

** 

0.0010 

Economically disadvantaged 

-0.3105 

0.1634 


0.0007 

Black 

-1.8786 

0.1855 

*** 

0.0184 

Hispanic 

-0.3075 

0.3226 


0.0002 

Asian 

0.1238 

0.6478 


0.0000 

American Indian/Alaskan Native 

-0.1430 

0.9617 


0.0000 

Hawaiian Native/Pacific Islander 

0.0000 

(omitted) 



Multiracial 

-1.4508 

0.4521 

*** 

0.0019 

Students receiving special education services 

-1.7553 

0.3039 

*** 

0.0061 

Rural 

-0.0063 

0.1560 


0.0000 

Female 

-0.4334 

0.1538 

** 

0.0015 

Grade 

-0.7795 

0.1106 

*** 

0.0090 

Attendance 

0.0269 

0.0046 

*** 

0.0061 

Year 

0.7930 

0.1325 

*** 

0.0065 


** significant at p < .01, *** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Number of course records = 5,482; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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Table C16. Exam score retest coefficients for North Carolina end-of-course exam, 
Physical Science, 2008/09-2010/11 

Variable 

Coefficient 

Standard 

error 

Significance 

Effect 

size 

NCVPS credit recovery Physical Science student 

-2.2940 

0.4747 

*** 

0.0035 

First Physical Science exam scaled score 

0.5607 

0.0141 

*** 

0.1953 

English learner student 

-1.3324 

0.4580 

** 

0.0013 

Economically disadvantaged 

0.2128 

0.1682 


0.0002 

Black 

-1.9462 

0.1832 

*** 

0.0169 

Hispanic 

-0.0039 

0.3798 


0.0000 

Asian 

0.7624 

0.8312 


0.0001 

American Indian/Alaskan Native 

1.5406 

1.0414 


0.0003 

Hawaiian Native/Pacific Islander 

0.0000 

(omitted) 



Multiracial 

0.9745 

0.5941 


0.0004 

Students receiving special education services 

-1.8633 

0.2029 

*** 

0.0127 

Rural 

-0.0922 

0.1686 


0.0000 

Female 

-0.2689 

0.1604 


0.0004 

Grade 

0.9829 

0.1018 

*** 

0.0140 

Attendance 

0.0209 

0.0032 

* * * 

0.0064 

Year 

-0.3099 

0.0942 

* * * 

0.0016 


** significant at p < .01, *** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Number of course records = 6,577; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. 

Source: Authors’ analysis of administrative data provided by the Education Research Data Center. 


Table C17. Exam score retest coefficients for North Carolina end-of-course exam, 
Biology, 2009/10-2011/12 

Variable 

Coefficient 

Standard 

error 

Significance 

Effect 

size 

NCVPS credit recovery Biology student 

-2.2981 

0.3256 

*** 

0.003 

First Biology exam scaled score 

0.5665 

0.0084 

*** 

0.199 

English learner student 

-2.5204 

0.2105 

*** 

0.008 

Economically disadvantaged 

-0.2367 

0.0926 

* 

0.000 

Black 

-1.7734 

0.1048 

*** 

0.015 

Hispanic 

-0.0332 

0.1796 


0.000 

Asian 

-0.0512 

0.4520 


0.000 

American Indian/Alaskan Native 

-1.3790 

0.4904 

** 

0.000 

Hawaiian Native/Pacific Islander 

0.8488 

2.3265 


0.000 

Multiracial 

-0.3020 

0.2814 


0.000 

Students receiving special education services 

-1.5127 

0.1083 

*** 

0.010 

Rural 

-0.0714 

0.0866 


0.000 

Female 

-0.2301 

0.0852 

** 

0.000 

Grade 

0.4572 

0.0527 

*** 

0.004 

Attendance 

0.0170 

0.0020 

*** 

0.004 

Year 

-0.1139 

0.0381 

** 

0.000 


* significant at p < .05, ** significant at p < .01, *** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Number of course records = 18,444; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. North Carolina administered Biology end-of- 
course exams in 2008/09, but NCVPS did not offer a Biology credit recovery option that year. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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Table C18. Exam score retest coefficients for North Carolina end-of-course exam, 
Civics and Economics, 2008/09-2010/11 

Variable 

Coefficient 

Standard 

error 

Significance 

Effect 

size 

NCVPS credit recovery Civics and Economics student 

-1.5980 

0.2620 

*** 

0.0024 

First Civics and Economics exam scaled score 

0.5973 

0.0084 

*** 

0.2416 

English learner student 

-2.0201 

0.2124 

*** 

0.0057 

Economically disadvantaged 

-0.4262 

0.0942 

*** 

0.0013 

Black 

-1.7772 

0.1060 

*** 

0.0175 

Hispanic 

0.0778 

0.1864 


0.0000 

Asian 

-1.3884 

0.3855 

*** 

0.0008 

American Indian/Alaskan Native 

-1.3732 

0.4132 

*** 

0.0007 

Hawaiian Native/Pacific Islander 

-5.7044 

2.6668 

* 

0.0003 

Multiracial 

-0.7231 

0.2936 

* 

0.0004 

Students receiving special education services 

-1.4141 

0.1086 

*** 

0.0106 

Rural 

-0.2882 

0.0888 

*** 

0.0007 

Female 

-0.6007 

0.0862 

*** 

0.0031 

Grade 

0.5911 

0.0561 

*** 

0.0070 

Attendance 

0.0176 

0.0021 

*** 

0.0043 

Year 

-0.4720 

0.0485 

*** 

0.0060 


* significant at p < .05, *** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Number of course records = 15,768; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 


Table C19. Exam score retest coefficients for North Carolina end-of-course exam, U.S. 


History, 2008/09-2010/11 

Variable 

Coefficient 

Standard 

error 

Significance 

Effect 

size 

NCVPS credit recovery U.S. History student 

-1.1505 

0.4301 

** 

0.0009 

First U.S. History exam scaled score 

0.6194 

0.0116 

*** 

0.2579 

English learner student 

-2.0739 

0.3825 

*** 

0.0035 

Economically disadvantaged 

-0.4058 

0.1356 

** 

0.0011 

Black 

-1.1635 

0.1497 

*** 

0.0073 

Hispanic 

-0.0538 

0.3024 


0.0000 

Asian 

1.2477 

0.6100 

* 

0.0005 

American Indian/Alaskan Native 

0.3617 

0.7166 


0.0000 

Hawaiian Native/Pacific Islander 

-0.8787 

5.7135 


0.0000 

Multiracial 

-0.2082 

0.4420 


0.0000 

Students receiving special education services 

-1.5188 

0.1789 

*** 

0.0087 

Rural 

-0.1445 

0.1313 


0.0001 

Female 

-0.8950 

0.1280 

*** 

0.0059 

Grade 

0.8250 

0.1289 

*** 

0.0049 

Attendance 

0.0160 

0.0026 

*** 

0.0044 

Year 

0.1485 

0.0726 

* 

0.0005 

* significant at p < .05, ** significant at p < .01, *** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Number of course records = 8,278; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. 


Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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Table C20. Exam score retest coefficients for North Carolina end-of-course exam, all 
courses combined, North Carolina Virtual Public School only, 2008/09-2011/12 


Variable 

Coefficient 

Standard 

error 

Significance 

Effect 

size 

First exam scaled score 

-0.5385 

0.0199 

*** 

0.2040 

English learner student 

-1.5269 

0.5401 

** 

0.0028 

Economically disadvantaged 

-0.4196 

0.2092 

* 

0.0014 

Black 

-0.6321 

0.2359 

** 

0.0025 

Hispanic 

0.4094 

0.4335 


0.0003 

Asian 

-0.5659 

1.2925 


0.0001 

American Indian/Alaskan Native 

-0.1070 

1.1768 


0.0000 

Hawaiian Native/Pacific Islander 

-5.8339 

5.2131 


0.0004 

Multiracial 

0.4605 

0.5223 


0.0003 

Students receiving special education services 

-1.2887 

0.2436 

*** 

0.0097 

Rural 

-0.0108 

0.2089 


0.0000 

Female 

0.0983 

0.1988 


0.0001 

Grade 

0.1748 

0.0935 


0.0012 

Attendance 

0.0202 

0.0048 

* * * 

0.0063 

Number of end-of-course exams failed first year 

-0.2151 

0.1380 


0.0008 

Number of end-of-course exams failed this year 

-1.8040 

0.0982 

* * * 

0.1056 

Year 

-0.7124 

0.1036 

*** 

0.0163 

* significant at p < .05, ** significant at p < .01, 

*** significant at p < .001. 




Note: Number of course records = 2,876; students take a retest only if they do not reach proficiency on first 
exam. Black includes African American; Hispanic includes Latino. 

Source: Authors’ analysis of administrative data provided by the Education Research Data Center. 
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Results of analyses of mid- and longer-term outcomes 


Table C21. Success in English II after English 1 credit recovery in North Carolina, 
2008/09-2011/12 

1 Variable 

Odds ratio 

Standard error 

Significance 1 

NCVPS credit recovery English 1 student 

0.9524 

0.0856 


Scaled score, English 1 

1.0194 

0.0026 

*** 

English learner student 

1.3035 

0.1102 

** 

Economically disadvantaged 

0.9977 

0.0382 


Black 

1.0461 

0.0421 


Hispanic 

0.8941 

0.0613 


Asian 

2.0070 

0.4441 

** 

American Indian/Alaskan Native 

1.1632 

0.1593 


Hawaiian Native/Pacific Islander 

0.1719 

0.2182 


Multiracial 

1.0101 

0.1082 


Students receiving special education services 

1.1172 

0.0463 

** 

Rural 

1.1437 

0.0404 

*** 

Female 

1.5293 

0.0558 

*** 

Grade 

1.7563 

0.0493 

*** 

Attendance 

1.0090 

0.0006 

*** 

Year 

0.9600 

0.0152 

* 


* significant at p < .05, ** significant at p < .01, *** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Success means that student earned credit for subsequent course. Number of students = 17,346. Black 
includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 


Table C22. Success in English III after English II credit recovery in North Carolina, 
2008/09-2011/12 

1 Variable 

Odds ratio 

Standard error 

Significance 1 

NCVPS credit recovery English II student 

0.9399 

0.0816 


Scaled score, English 1 

1.0184 

0.0036 

*** 

English learner student 

1.1399 

0.1496 


Economically disadvantaged 

0.9985 

0.0461 


Black 

1.0568 

0.0533 


Hispanic 

0.9388 

0.0756 


Asian 

0.7388 

0.1582 


American Indian/Alaskan Native 

2.1235 

0.3757 

*** 

Hawaiian Native/Pacific Islander 

0.5759 

0.7119 


Multiracial 

0.9019 

0.1083 


Students receiving special education services 

0.9397 

0.0576 


Rural 

0.9177 

0.0411 


Female 

1.3983 

0.0630 

*** 

Grade 

1.6372 

0.0495 

*** 

Attendance 

1.0100 

0.0008 

*** 

Year 

0.9835 

0.0245 



*** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Success means that student earned credit for subsequent course. Number of students = 11,845. Black 
includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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Table C23. Success in English IV after English III credit recovery in North Carolina, 
2008/09-2011/12 

1 Variable 

Odds ratio 

Standard error 

Significance j 

NCVPS credit recovery English III student 

0.9049 

0.1197 


Scaled score, English 1 

1.0045 

0.0057 


English learner student 

1.0170 

0.2252 


Economically disadvantaged 

0.8707 

0.0659 


Black 

1.2199 

0.1038 

* 

Hispanic 

1.1393 

0.1494 


Asian 

1.0382 

0.3484 


American Indian/Alaskan Native 

0.7427 

0.1888 


Hawaiian Native/Pacific Islander 

1.0000 

(omitted) 


Multiracial 

0.8460 

0.1556 


Students receiving special education services 

1.1138 

0.1241 


Rural 

1.1349 

0.0831 


Female 

1.3927 

0.1057 

*** 

Grade 

1.3296 

0.0429 

*** 

Attendance 

1.0081 

0.0011 

*** 

Year 

1.1977 

0.0690 

** 


* significant at p < .05, ** significant at p < .01, *** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Success means that student earned credit for subsequent course. Number of students = 7,840. Black 
includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 


Table C24. Success in Algebra II or Geometry after Algebra 1 credit recovery in 

North Carolina, 2008/09-2011/12 

1 Variable 

Odds ratio 

Standard error 

Significance I 

NCVPS credit recovery Algebra 1 student 

0.8774 

0.0863 


Scaled score, Algebra 1 

1.0460 

0.0023 

*** 

English learner student 

1.0654 

0.0875 


Economically disadvantaged 

0.8922 

0.0297 

*** 

Black 

0.9158 

0.0337 

* 

Hispanic 

0.9058 

0.0535 


Asian 

1.0985 

0.1733 


American Indian/Alaskan Native 

1.1183 

0.1770 


Hawaiian Native/Pacific Islander 

0.7115 

0.6621 


Multiracial 

0.8104 

0.0685 

* 

Students receiving special education services 

1.0055 

0.0446 


Rural 

1.0743 

0.0335 

* 

Female 

1.2970 

0.0398 

*** 

Grade 

1.0073 

0.0007 

*** 

Attendance 

1.3010 

0.0262 

*** 

Year 

0.9885 

0.0153 



* significant at p < .05, *** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Success means that student earned credit for subsequent course. Number of students = 20,167. Black 
includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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Table C25. Success in subsequent course in sequence, all courses combined, 

North Carolina Virtual Public School only, 2008/09-2011/12 

1 Variable 

Odds ratio 

Standard error 

Significance 1 

Scaled score, English 1 or Algebra 1 

1.0185 

0.0065 

** 

English learner student 

0.9161 

0.2113 


Economically disadvantaged 

1.0015 

0.0778 


Black 

1.2598 

0.1121 

** 

Hispanic 

1.0077 

0.1445 


Asian 

0.8726 

0.3552 


American Indian/Alaskan Native 

0.6642 

0.2221 


Hawaiian Native/Pacific Islander 

1.0000 

(omitted) 


Multiracial 

1.2425 

0.2479 


Students receiving special education services 

0.9963 

0.0975 


Rural 

1.1299 

0.0852 


Female 

1.3878 

0.1055 

*** 

Grade 

1.5086 

0.0615 

*** 

Attendance 

1.0074 

0.0015 

*** 

Number of end-of-course exams failed first year 

1.0054 

0.0553 


Number of end-of-course exams failed this year 

0.8350 

0.0353 

*** 

Year 

0.9593 

0.0423 



** significant at p < .01, *** significant at p < .001. 

Note: Success means that student earned credit for subsequent course. Number of students = 3,571. Black 
includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 


Table C26. Re-enrollment in school year following credit recovery in North Carolina, 
2008/09-2011/12 

1 Variable 

Odds ratio 

Standard error 

Significance 1 

NCVPS credit recovery student 

1.5909 

0.0633337 

*** 

English learner student 

0.6758 

0.0330 

*** 

Economically disadvantaged 

0.7196 

0.0151 

*** 

Black 

1.3940 

0.0311 

*** 

Hispanic 

1.3478 

0.0535 

*** 

Asian 

1.5830 

0.1507 

*** 

American Indian/Alaskan Native 

1.5079 

0.1303 

*** 

Hawaiian Native/Pacific Islander 

1.4344 

0.7492 


Multiracial 

1.2413 

0.0701 

*** 

Students receiving special education services 

0.7663 

0.0189 

*** 

Rural 

1.0299 

0.0203 


Female 

1.1428 

0.0225 

*** 

Grade 

0.7380 

0.0079 

*** 

Attendance 

1.0408 

0.0003 

*** 

Year 

0.4013 

0.0043 

*** 


*** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Number of students = 178,232. Black includes African American; Hispanic includes Latino. 
Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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Table C27. Graduation after completion of at least one credit recovery course in 
North Carolina 

1 Variable 

Odds ratio 

Standard error 

Significance 1 

NCVPS credit recovery student 

0.6704 

0.0295 

*** 

English learner student 

0.6292 

0.0359 

*** 

Economically disadvantaged 

0.6599 

0.0147 

*** 

Black 

1.6172 

0.0390 

*** 

Hispanic 

1.3160 

0.0560 

*** 

Asian 

2.2554 

0.2305 

*** 

American Indian/Alaskan Native 

1.3729 

0.1307 

*** 

Hawaiian Native/Pacific Islander 

1.4244 

0.7508 


Multiracial 

1.0665 

0.0666 


Students receiving special education services 

0.6676 

0.0188 

*** 

Rural 

0.8409 

0.0179 

*** 

Female 

1.4214 

0.0301 

*** 

Grade 

4.8377 

0.0479 

*** 

Attendance 

1.0305 

0.0003 

*** 

Year 

1.9207 

0.0192 

*** 


*** significant at p < .001. 

NCVPS is North Carolina Virtual Public School. 

Note: Number of students = 122,307. Black includes African American; Hispanic includes Latino. 
Source: Authors' analysis of administrative data provided by the Education Research Data Center. 


Table C28. On-time graduation after completion of at least one credit recovery 
course in North Carolina 


1 Variable 

Odds ratio 

Standard error Significance I 

NCVPS credit recovery student 

1.4033 

0.0459 

*** 

English learner student 

0.4677 

0.0241 

*** 

Economically disadvantaged 

1.2983 

0.0217 

*** 

Black 

0.8633 

0.0156 

*** 

Hispanic 

1.2528 

0.0373 

*** 

Asian 

1.0278 

0.0650 


American Indian/Alaskan Native 

1.3406 

0.0855 

*** 

Hawaiian Native/Pacific Islander 

1.7777 

0.5589 


Multiracial 

1.2168 

0.0536 

*** 

Students receiving special education services 

0.9742 

0.0232 


Rural 

1.0553 

0.0165 

*** 

Female 

1.0476 

0.0161 

** 

Grade 

0.7132 

0.0069 

*** 

Attendance 

1.0065 

0.0004 

*** 

** significant at p < .01, *** significant at p < .001. 




NCVPS is North Carolina Virtual Public School. 




Note: On-time graduation means graduation within four years of entering high school as a grade 9 student. 

Year variable excluded because more than 95 percent of viable observations were for 2011/12. Number of 
students = 94,828. Black includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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Table C29. Re-enrollment in school year following credit recovery, North Carolina 
Virtual Public School only, 2008/09-2011/12 

1 Variable 

Odds ratio 

Standard error 

Significance | 

English learner student 

0.7087 

0.1658 


Economically disadvantaged 

0.6999 

0.0595 

*** 

Black 

1.4754 

0.1406 

*** 

Hispanic 

1.7552 

0.2915 

*** 

Asian 

1.1092 

0.5087 


American Indian/Alaskan Native 

1.3689 

0.6858 


Hawaiian Native/Pacific Islander 

1.0000 

(omitted) 


Multiracial 

1.6034 

0.3591 

* 

Students receiving special education services 

0.7855 

0.0845 

* 

Rural 

0.8330 

0.0693 

* 

Female 

1.0566 

0.0870 


Grade 

0.6181 

0.0277 

*** 

Attendance 

1.0352 

0.0013 

*** 

Number of end-of-course exams failed first year 

0.6301 

0.0331 

*** 

Number of end-of-course exams failed this year 

1.5197 

0.0764 

*** 

Year 

0.2233 

0.0138 

*** 


* significant at p < .05, *** significant at p < .001. 

Note: Number of students = 12,854. Black includes African American; Hispanic includes Latino. 
Source: Authors' analysis of administrative data provided by the Education Research Data Center. 


Table C30. Graduation after completion of at least one credit recovery course, 

North Carolina Virtual Public School only 

1 Variable 

Odds ratio 

Standard error 

Significance 1 

English learner student 

0.6986 

0.1719 


Economically disadvantaged 

0.7107 

0.0594 

*** 

Black 

1.8740 

0.1770 

*** 

Hispanic 

1.4122 

0.2312 

* 

Asian 

1.6038 

0.8081 


American Indian/Alaskan Native 

1.1095 

0.6358 


Hawaiian Native/Pacific Islander 

1.0000 

(omitted) 


Multiracial 

1.2953 

0.2917 


Students receiving special education services 

0.7770 

0.0863 

* 

Rural 

0.9959 

0.0828 


Female 

1.3003 

0.1047 

*** 

Grade 

3.4822 

0.1293 

*** 

Attendance 

1.0244 

0.0011 

*** 

Number of end-of-course exams failed first year 

1.1504 

0.0561 

** 

Number of end-of-course exams failed this year 

0.6922 

0.0330 

*** 

Year 

1.7785 

0.0832 

*** 


* significant at p < .05, ** significant at p < .01, *** significant at p < .001. 

Note: Number of students = 6,421. Black includes African American; Hispanic includes Latino. 
Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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Table C31. On-time graduation after completion of at least one credit recovery 
course, North Carolina Virtual Public School only 

1 Variable 

Odds ratio 

Standard error 

Significance | 

English learner student 

1.0599 

0.2874 


Economically disadvantaged 

1.3331 

0.0968 

*** 

Black 

1.2523 

0.1014 

** 

Hispanic 

1.2906 

0.1690 


Asian 

1.1461 

0.4253 


American Indian/Alaskan Native 

1.4541 

0.5608 


Hawaiian Native/Pacific Islander 

1.0000 

(omitted) 


Multiracial 

1.6842 

0.2968 

** 

Students receiving special education services 

1.3446 

0.1430 

** 

Rural 

0.9095 

0.0638 


Female 

1.3005 

0.0886 

*** 

Grade 

0.7808 

0.0323 

*** 

Attendance 

1.0088 

0.0014 

*** 

Number of end-of-course exams failed first year 

0.7651 

0.0338 

*** 

Number of end-of-course exams failed this year 

0.2110 

0.0205 

*** 


** significant at p < .01, *** significant at p < .001. 

Note: On-time graduation means graduation within four years of entering high school as a grade 9 student. 
Year variable excluded because more than 95 percent of viable observations were for 2011/12. Number of 
students = 4,744. Black includes African American; Hispanic includes Latino. 

Source: Authors' analysis of administrative data provided by the Education Research Data Center. 
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Appendix D. Interview protocols for third-party vendors 


This appendix includes the telephone interview protocol used to collect data for the 
descriptions of each of the seven third-party credit recovery vendor programs. 

Consent to participate in interview and introduction 

Good [MORNING/AFTERNOON]. My name is [NAME], I am a [RESEARCH ASSO- 
CIATE/RESEARCH SCHOLAR] at the Friday Institute for Educational Innovation at 
North Carolina State University. 

We are conducting a study of the North Carolina Virtual Public School’s online credit 
recovery program for high school students. 

The reason we have asked you to talk to us today is because, according to our records, at 
least one North Carolina school district has enrolled students in courses offered by your 
organization as a means of helping them to recover credit for a course or courses they had 
failed. Consequently, we believe that you can help us understand more about the context 
of online credit recovery in North Carolina. 

During this short interview, I will ask you to share with us some basic information about 
your organization and the services you currently provide or have provided in the recent 
past to North Carolina school districts. 

Your participation in this study is voluntary. Because your name will not be associated 
with any of the information you provide, there is minimal risk to you as a result of partic- 
ipating in this study. In addition, whether you decide to participate or not will not affect 
your organization’s right to offer services to North Carolina school districts. 

The information in the study records will be kept confidential to the full extent allowed 
by law. Your responses will not be linked in any way to your name. In addition, no direct 
reference will be made in oral or written reports that could link you to the study. 

Questions 

1. Since the 2008/09 school year, how many North Carolina school districts have enrolled 
students in courses your organization offers, with the explicit intent of helping those 
students recover credit for courses they previously failed? 

a. [IF ABLE TO DISTINGUISH USE OF COURSES FOR CREDIT RECOVERY] 
Can you provide for us either an exact or estimated count per school district of the 
number of students enrolled in those courses for each school year from 2008/09 
through 2011/12? 

b. [IF UNABLE TO DISTINGUISH USE OF COURSES FOR CREDIT RECOV- 
ERY] If you are willing to do so, can you estimate how many of those enrollments 
were for credit recovery, per school district, for each school year from 2008/09 
through 2011/12? 


D-l 


2. In which of your courses did North Carolina students enroll between 2008/09 and 

2011 / 12 ? 

3. During the time period in question, did your organization’s approach to online learn- 
ing require student participation for an entire semester/year in order to earn credit, or 
was credit based on mastery, regardless of length of participation? 

4. During the time period in question, were your courses student-driven or teacher-led? In 
other words, did students set their own learning pace, or did they follow a schedule set 
by an online teacher? 

5. During the time period in question, were your course offerings synchronous, asynchro- 
nous, blended (i.e., a mix of synchronous and asynchronous), or some mix of some or 
all of these? 

6. During the time period in question, what was your organization’s typical per-student, 
per-course charge to a district for enrollment in your courses? 
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Notes 


1. Before the 2011/12 school year, NCVPS courses were provided at no cost to districts. 
During the 2011/12 school year the NCVPS funding formula was changed to require 
school districts to return to the state part of their fiscal year allotments equivalent to 
the cost of educating each of their students enrolled in NCVPS courses (North Caro- 
lina General Assembly, 2011). Previously, student enrollment in NCVPS classes largely 
did not affect district funding. Because of this change, individual district responses 
in terms of students allowed to enroll in NCVPS courses (including credit recovery 
courses) varied, making student enrollment patterns in credit recovery in some dis- 
tricts after 2011/12 very different from enrollment in previous years. Because the full 
impact of the change on enrollment is not yet clear, the study limited data analysis 
to the years preceding the change. As a clearer understanding of the impact of the 
change across districts and student groups on credit recovery enrollment is developed, 
researchers can reconsider inclusion of data from later years. 

2. Because of limitations in the ability to identify all third-party online credit recovery 
students (see box A1 in appendix A), the study does not report demographic infor- 
mation about those students, as the identifiable students may not be demographically 
representative of all third-party online credit recovery students. These students are, 
however, included in the analyses for research questions 2 and 3 (which combine data 
for all identifiable non-NCVPS credit recovery students), as they are clearly identifi- 
able as participants in at least one of the non-NCVPS credit recovery programs. 

3. Most third-party vendors do not enroll students in a program but rather offer full 
access to their products with limitations tied to individual students’ scope of use. 
Schools typically have the option to purchase access either for a single student or for 
multiple students (that is, school-wide access), with those students then able to access 
a certain number of resources (up to four courses) at a given time. Some vendors allow 
full access to all their products without limitations. 

4. Of the variables considered in these deeper analyses, the most reliable predictor of a 
student’s achievement level or scaled score growth on the retest was her or his score 
the first time she or he took the exam (in other words, where she or he started academ- 
ically before credit recovery) — not participation in a specific credit recovery option 
(see tables C11-C19 in appendix C). 
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